Ensure data is identical between collective members
As a PI Administrator, I need to trust that the data is identical between my collective members. Currently, there is no way for me to be sure it is the same unless I do a side by side comparison. Data can be missing, or slightly off, and I have no easy way to understand this right now.
ADMIN RESPONSE
Aug 20, 2022
The architectural design of a PI Collective is such that there is no data synchronization between the collective members. We have established procedures to sync the data using PI to PI Interface. Is this a viable solution?
No, using a PI to PI interface to synchronize data between members within a collective is the wrong approach. We have the concept of primary/secondary members in the collective already; however, the primary member needs to be the write-only member, where the rest of the nodes are read-only (Example: need to get point configuration or data? You'll get it from the secondaries, although you can choose to grab it from the primary depending on the times sensitivity of the data pull). However, the primary member should not be static like it currently is. The primary member would need to be dynamic and automatically shift as the primary goes down.
It should never be the responsibility of the data provider to ensure all members of a collective have the data they need. Example, review the MongoDB system to see how they're able to achieve the HA requirements and data integrity across members. If you look at their concept of sharding, that would likely be a similar approach required & currently used to section data, but in this case - by timestamp.
Sync data using PI to PI Interface is not always a viable solution. It is required to be able get an advice if data are missing in one member of a collective. A health point making sur that all collective member have the same information would be helpful.
PI to PI works to send data to another collective member when buffering is not or cannot be used. However the big hurdle for us at least with some of our manual entry data is that there is no way to delete from a secondary member without adding custom code. It would be nice if PI to PI or something else could handle the deletion of data.
How does one know if one collective member has stopped receiving data from PI to PI? I guess the question is how do we ensure to our users that data is the same no matter what collective member they connect to? I understand your design, but seems like there needs to be a way then for PI to PI to verify data is arriving on all nodes, if not, back-fill or fix data to the single nodes that is impacted.
Since this was posted, we now have the PI connector. We would like to do more testing with that, but a bug took it back away from us earlier. Seems that the bug is fixed and well do some testing now.
No, using a PI to PI interface to synchronize data between members within a collective is the wrong approach. We have the concept of primary/secondary members in the collective already; however, the primary member needs to be the write-only member, where the rest of the nodes are read-only (Example: need to get point configuration or data? You'll get it from the secondaries, although you can choose to grab it from the primary depending on the times sensitivity of the data pull). However, the primary member should not be static like it currently is. The primary member would need to be dynamic and automatically shift as the primary goes down.
It should never be the responsibility of the data provider to ensure all members of a collective have the data they need. Example, review the MongoDB system to see how they're able to achieve the HA requirements and data integrity across members. If you look at their concept of sharding, that would likely be a similar approach required & currently used to section data, but in this case - by timestamp.
Sync data using PI to PI Interface is not always a viable solution. It is required to be able get an advice if data are missing in one member of a collective. A health point making sur that all collective member have the same information would be helpful.