Dacs Deamon not connecting

jeff.waterman

One thing I would like to discuss, is the DACS daemon connection that isn’t working quite right: I have two instances of the application connecting to the DACS daemon (because all Osiris components are in a hot-hot standby distributed over two data centers) and it seems the second instance to start often has a hard time connecting to the DACS daemon right away. I originally had the application configured with a six-second connect timeout, which failed often, and found that it sometimes takes up to minute while at other times it works right away:

2018-12-28 02:29:53,590 [DacsProvider] Waited since 28.12.2018 02:29:47 for AuthorizationConnectionUp, but at 28.12.2018 02:29:53 the DACS daemon connection state is still AuthorizationConnectionDown.

2018-12-28 02:30:02,087 [DacsProvider] Waited since 28.12.2018 02:29:56 for AuthorizationConnectionUp, but at 28.12.2018 02:30:02 the DACS daemon connection state is still AuthorizationConnectionDown.

2018-12-28 03:30:08,682 [DacsProvider] DACS daemon connection is AuthorizationConnectionUp at 28.12.2018 03:30:08 after waiting since 28.12.2018 03:30:08.

2018-12-28 12:18:24,727 [DacsProvider] DACS daemon connection is AuthorizationConnectionUp at 28.12.2018 12:18:24 after waiting since 28.12.2018 12:18:24.

2018-12-28 13:15:42,420 [DacsProvider] Waited since 28.12.2018 13:15:36 for AuthorizationConnectionUp, but at 28.12.2018 13:15:42 the DACS daemon connection state is still AuthorizationConnectionDown.

2018-12-28 13:15:50,045 [DacsProvider] Waited since 28.12.2018 13:15:44 for AuthorizationConnectionUp, but at 28.12.2018 13:15:50 the DACS daemon connection state is still AuthorizationConnectionDown.

2018-12-28 13:23:25,543 [DacsProvider] DACS daemon connection is AuthorizationConnectionUp at 28.12.201813:23:25 after waiting since 28.12.201813:22:24.

2018-12-28 13:24:28,175 [DacsProvider] DACS daemon connection is AuthorizationConnectionUp at 28.12.2018 13:24:28 after waiting since 28.12.2018 13:24:27.

2018-12-28 13:25:47,831 [DacsProvider] DACS daemon connection is AuthorizationConnectionUp at 28.12.2018 13:25:47 after waiting since 28.12.2018 13:25:47.

2018-12-28 13:28:43,447 [DacsProvider] DACS daemon connection is AuthorizationConnectionUp at 28.12.2018 13:28:43 after waiting since 28.12.2018 13:28:42.

Any pointers why this is? Does it have to do with the load-balancer at pstestmddist.uk.hibm.hsbc:8211? Times above are GMT+1, in case you want to cross-reference with the DACS daemon logs. What would be a reasonable timeout for the connection?

zoya faberov

Hello @jeff.waterman,

I have tested with two instances of Open DACS Java DacsInfoServices example (very compact, connects to the specified daemon and prints available services info), connecting one after the other, multiple times, and timestamping the attempts.

Both instances took about the same time to connect, a couple of seconds, consistently, at no time either of them have taken over 6 seconds to connect.

I think it suggest the cause is likely to be within local environment, I think the load balancer is a good guess.

I would try connecting explicitly, directly, bypassing the load balancer.

Another thought is that dacs daemons limit the number of clients that can attach simultaneously with DACS_MAX_SNKD_CLIENTS

It may be worth checking with your market data team what your daemon maxes out at, and also what other apps may be connecting simultaneously and taking up some of the available client connections.