How to decode the encSetDefs?
Have tried to decode it as a container indicated by containerType and to decode it as Series but failed. Could you tell us which API should be used to decode it? And when encoding, what kind of structure should be encoded into it?
Set definitions are encoded using the set definition encoding format (so they are not communicated as a field list or any other container type but they are their own specific ncoding format to optimally represent definitions on the wire). Essentially, the encoded set definitions would be either a Field Set Definition (if it is defining layout for Field List content) or an Element Set Definition (if it is defining layout for Element List content). The type of this is generally determined by the container type specified on the construct providing the encSetDefs. For example, you are processing a Series that has set definitions present. If the series.containerType indicates RSSL_DT_FIELD_LIST, this tells you that the encoded set definitions are Field Set Defs and you would use the Field Set Def encoding/decoding methods when processing. If it said RSSL_DT_ELEMENT_LIST, this would be element set defs and you would use element set def methods.
There is an entire section in the UPA Developers Guide dedicated to explaining this in much more detail. It has tables that detail each method used for encoding or decoding set definitions as well as code examples that show how to use these concepts. This should help give you more information to get your code up and running. The chapter you want to look at is Chapter 10.6 Set Definitions and Set Defined Data. This chapter and its subsections cover all things related to set definitions.