Enums and ms timestamp fields in Workspace Realtime API

davidk · June 16

Hi— we had been debugging the new lseg.data library realtime APIs on a system that had been running Eikon. When running eikon Eikon, fields with enums (like IMB_TYPE) and fields with timestamps (like IMB_TIM_MS and QUOTIM_MS) were all coming back as integers using the lseg.data streaming APIs, for example using the following code:

import lseg.data.content.pricing as pricing
pricing_stream = pricing.Definition(universe=['AAPL.O'], fields=['BID', 'ASK', 'QUOTIM_MS']).get_stream()
stream.get_snapshot()

# QUOTIM_MS used to be an integer, but now is a string like '20:40:42.605'

pricing_stream = pricing.Definition(universe=['AAPL.ITC'], fields=['IMB_TYPE', 'IMB_SIDE', 'IMBTIM_MS']).get_stream()
stream.get_snapshot()
# All fields used to be integers, now they are strings.

We have now cut over to Workspace for the front end, but it seems that now these fields are all coming back with text. Is this a setting that was possibly flipped when the upgrade occurred, or is this an expected change with the change of front-end? We were taken aback when the front-end update affected the API output.

Jirapongse · June 17

@davidk

Thank you for reaching out to us.

Yes, you are correct. The Workspace platform returns expanded strings for enumeration fields and converts quote times into string format. According to the log, this should happen on the workspace platform or streaming server, not in the library.

[2025-06-17T10:16:07.105445+07:00] - [sessions.desktop.workspace.0] - [DEBUG] - [9904 - Msg-Proc-ThreadOMMSTREAMING_PRICING_0.0] - [omm_stream_connection] - [_process_message] - [OMMSTREAMING_PRICING_0.0] process message {"Fields": {"BID": 197.08, "ASK": 197.2, "QUOTIM_MS": "23:59:55.038", "CURRENCY": "USD"}, "State": {"Data": "Ok", "Stream": "NonStreaming", "Code": "None", "Text": "All is well"}, "Qos": {"Rate": "TimeConflated", "Timeliness": "DelayedUnknown"}, "Type": "Refresh", "SeqNumber": 64702, "ID": 7, "Key": {"Name": "/AAPL.O", "NameType": "Ric", "Service": "Q"}}

I also tested it by consuming from the real-time streaming server (Real-Time Optimized). The output looks like this:

        "Fields": {

            "CURRENCY": "USD",

            "BID": 197.080000,

            "ASK": 197.200000,

            "QUOTIM_MS": 86395038

        }

The enumeration field is expanded but the value in the QUOTIM_MS field is still an integer.

I understand that the string value of the "QUOTIM_MS" field may be specific on the workspace.desktop session.

We need to contact the Workspace support team to verify this behavior. I have submitted a case on your behalf. The case number is 14891197.

Jirapongse · June 26

@davidk

The product team confirmed that this is an expected behavior of the workspace desktop session.

I believe this change makes the data easier to understand for desktop users; however, it will affect migrated users who subscribe to those fields from the desktop session.

For enum fields, you can add the following extended_params parameter to get the enum as integers.

pricing_stream = pricing.Definition(
    universe=['AAPL.O'], 
    fields=['BID', 'ASK', 'QUOTIM_MS','CURRENCY'],
    extended_params={"Key": {"ExpandEnum": False}}).get_stream()
pricing_stream.open(with_updates=False)
pricing_stream.get_snapshot()

For the _MS field, you can implement a function to convert it values back to integers. For example:

import re
from datetime import datetime

isoTimeRegex = r'^(?:[01]\d|2[0-3]):[0-5]\d:[0-5]\d(?:\.\d{1,3})?$'

def convert_timestr_to_int(time_str):
   if isinstance(time_str, str) and re.match(isoTimeRegex, time_str):
       to = datetime.strptime(time_str, "%H:%M:%S.%f")
       return (to.hour * 3600 + to.minute * 60 + to.second) * 1000 + int(
           to.microsecond / 1000
       )
   return time_str

Then, you can use the function like this:

df = pricing_stream.get_snapshot()
df["QUOTIM_MS"] = df["QUOTIM_MS"].apply(convert_timestr_to_int)

However, if you consume data from other platforms, such as RTDS, and RTO, you may receive enum values as integers (depending on the servers' configurations) and the _MS fields as integers.

davidk · June 26

Thanks very much Jirapongse, is there any documentation of the meanings of any other extended_params that may exist? Had we known about that it could have saved a lot of headache.

Jirapongse · June 27

@davidk

I couldn't find this parameter on the WebSocket API Specifications and Developers Guide.

This parameter is shared by the development team on a case-by-case basis to help mitigate the issue.

pf · June 27

Hi @davidk,

If you refer to LSEG Data Library for Python - Reference Guide, you'll find extended_params in pretty all Definition classes (not only pricing.Definition):

This API concep allows to inject any additional and/or future new parameter(s) into the json request body. Especially for the lseg.data.ipa API which evolves often (but not only).

There isn't documentation because it was designed with this way to allow forward compatibility with future new parameters of the underlying back-end (in this case, the Workspace streaming layer which is in charge to connect to LSEG infra and retrieve realtime data).

Currently, for the pricing API, I don't see other parameter than ExpandEnums that could be injected in request.

davidk · June 27

Thanks Jirapongse and pf.

I believe it is clear that ExpandEnums should be (and should have been) documented somewhere. Had we known about this two weeks ago when I raised this issue, it would have greatly helped my production fire. Had it been available and documented when I first raised this issue 6 months ago as this transition began (link below) I could have avoided production transition heartache entirely, but it does not seem this parameter's existence was even clear internally to LSEG.

Please, I encourage you, document all the settings one can put in the fields (perhaps it is only one for now, but as you note, that could change), and within the library documentation link to where that information can be found. Simply having an "extended parameters" field does no one any good if no one inside or outside the company knows how to use it.

Similarly, I think it would be wise to avoid, and at least document, any cases where the exact same API call, as a user gives different answers based where iit is used. The behavior of a function shouldn't depend on the particular terminal setup a user is coming from.From the above discussion it appears there is different behavior of the exact same API calls, depending on if you're in Workspace, in Codebook, using RDP, etc. all using the same inputs. This makes it very difficult to share code, diagnose issues, etc.

https://community.developers.refinitiv.com/discussion/131391/requesting-us-equity-imbalance-type-side-via-workspace-api-numeric-data-for-string-fields

Enums and ms timestamp fields in Workspace Realtime API

Answers

Categories

EXPLORE OUR SITES

Enums and ms timestamp fields in Workspace Realtime API

Answers

Categories