Building a OIP record ecosystem
OIP records are encoded with protobuff (v3) messages. These messages are like metadata schema that contains fields and specifications of the type of data that can fit in each field. Every schema of a valid OIP record type must be published in the blockchain prior to its use and it can be used by anyone.
Here we will define the first OIP record types and will explain the hierarchy between them.
Primitives
Here we define 2 primitive record types: profile and relationship. All record types must belong to either these two primitive types.
profile
Type of record to store metadata of things.
This type has four fields only: name, description, validRelationships and notes. They are mostly self-explanatory, but one requires more digging.
Briefly, validRelationships is cross-validation that a relationship in the chain is accepted by all parties. For example, in a relationship of employment, the company issues the employment record and the employee, to accept the job, could add that record identifier to the list of validRelationships. We will be talking more about this in the future.
relationship
Type of records to store metadata of relationships between records of the type profile.
This primitive record has the basic elements of a relationship: name, description of the relationship, since to indicate when the relationship started and two other fields: parties and roles.
The field parties list all the profiles involved in the relationship. The field roles is a list in the same order as parties with the role of each profile listed in parties.
We will talk more about relationships later.
Hierarchy of record types
To build a new record type, we must first define what is its parent type. The OIP record parent type can be any existing record type already published.
Because OIP uses protobuff (v3) to encode the information in the chain we cannot extend the fields of a parent type automatically. Instead, every field in the message of the new type must be reinstated. However, we use the commented line with a specific format in the message to let OIP know the parent type:
// parentType: profile
The new OIP record type must be compatible with its parent and this has implications to how we will write the new message. To make the message compatible, we need to follow the protobuff guidelines.
First, every field in the parent type must be present in the exact form and number in the child type.
Renaming an existing field just because of semantics is not a problem, protobuff will not break, neither the OIP.
Adding new fields are also not a problem, just never change the field number of the parent type.
Also, deleting a field is not a problem. Just make sure to mark the field number as reserved to avoid conflict. You can do that using the keyword reserved.
What happens if I reuse a number?
The data encoded with this type won't be correctly decoded by the protobuff message of the parent type. Thus, because OIP will skip the error silently, the consequence is that records posted using this record type won't show in searches on the parent type - which is the whole reason why we use hierarchy between records.
If we need to significantly change a record type, we should consider to fork it instead of parent it. In forking a profile we only need to make our type compatible with the parent type of the profile we fork.
Designing DDEMd records
DDEMd is a distributed database of electron micrographs datasets that are generated by assays on the electron microscope taking several electron micrographs. The most fundamental part of this database is each individual electron micrograph, which in turn it is a particular type of image.
Record type profile.image
We decided to build a record type for images, as simple as possible for now. Images are things so we use profile as the parent type and we add just two more fields: takenBy and date to say who took the image and when the image was produced.
Electron micrographs are a particular type of image that has some peculiar metadata associated with it. So we will build the record type electronMicrograph instead of specifying a field type in image.
A general rule of thumb to use the OIP as we design is to make a child type every time we feel tempted to use the field type without a limited number of choice. For example, blood type is ok, an image of the type electronMicrograph isn't.
Another rule of thumb is that when designing a record type that you think it might be useful to others, make it as simple as possible so more people can use it as a parent type. Don't worry about namespace because in real life each record has its own unique identifier once is placed in the blockchain and it is part of its name.
Here we just named them by the human-readable name "part", but in reality the record type we call image is named image_9243fcc2d540 where 9243fcc2d540 is the first 15 digits of the transaction id that published the image record schema.
Also, we sometimes refer to a record type by the immediate parent type followed by the type's name. For example: image.electronMicrograph. Also we may extend to include further parents: profile.image.electronMicrograph depending on what we want to call the readers attention to.
Record type image.electronMicrograph
The new fields in electronMicrographs is takenWith, magnification and defocus. The latter two fields are specific to electron micrographs, the third one is to describe the instrument that took the image.
The takenWidth field should be filled with the identifier for the profile of the electron microscope that took the electron micrograph.
For that, we need to make the microscope's profile.
Record type microscope
microscopebefore we do that, we should first search for a parent type that makes sense. Microscopes are devices with a specific function. Thus, we should first search for a parent type that describes devices. Since this is the first project, we will make the record type device
Record type profile.device
We parent profile and add three fields to this record type: serial, manufactures and parts.
The field manufactures takes identifiers of the companies or groups involved in making the device. We will talk about the record type company very soon.
The field parts takes identifiers of the records of the type part. We are not going through that rabbit hole now, so we will leave it as is and hope that someday, someone will build a record type part to describe components of devices.
Now we can build the record type microscope.
Record type devices.microscope
A microscope is a type of device, that is it. So no fields added.
An electron microscope is a type of microscope, so we build microscope.electron
Record type microscope.electron
The same thing as previously but this opens a space for a remark about hierarchy.
We want someone searching for all devices to be able to find a record that describes an electron microscope. But we don't want them to find the record if they are searching for cameras. Some microscopes have cameras but aren't cameras. Thus, camera is a sister record type of microscope both with the same device parent type.
Obviously, there are multiple types of microscopes and if someone is searching for a confocal microscope, we don't want OIP to list electron microscopes.
This is why we make new record types referencing parent types, this will improve the speed of searches within a certain scope.
Ok, now that we can safely fill the field takenWith on the record electronMicrograph, we will continue to design our schema.
Record type assay
As a scientific database, DDEMd is not just a set of images, they were taken by specific samples, using certain protocols and etc. We could put all this info in each electronMicrograph but then we thought it would be hard to make it obvious that a subset of these images was collected together, in a single assay.
For this reason, we built the record type assay to hold the metadata of scientific assays. assay is a child type of profile and add the following fields:
samples, protocols, executeBy and data. Those are things that should be in any assay.
Before we go on explaining these fields, obviously we are not interested in posting a record of the generic assay type, we want to make electronMicroscopy assays. As we did before we just build electronMicroscopy as a copy of assay that parents assay.
Record type assay.electronMicroscopy
A few comments here.
First, the field data is where we will put all the identifiers of each of the records of the type image.electronMicrograph that was produced as a result of this assay.
Then we will build the record type protocol to hold information about the protocol used to take these images. Then we will build sample that will be used to publish records that hold information about the sample: where it was collected, by who, when and etc.
We will save the reader from going in the details of all the other record types involved in here, but it gives a segway for us to talk about relationships between records of type profile.
Relationships between OIP records
We just say that records of the type assay need to include who executed the assay.
In this field, we should reference the people who were involved in making the assay. For that reason, we created the record type profile.people to be used to publish metadata about individual people.
Also, this field should hold identifiers of laboratories, universities, companies and any other institution responsible for the assay.
Notice again the pattern here: laboratories, universities, and companies are all institutions.
So we created the profile.institution record type and also the types profile.institution.laboratory, profile.institution.university and profile.institution.company.
Now we can finally talk about relationships.
Say that Bob's is working on his Ph.D. under the supervision of Prof. Alice at the ACME University.
So Alice is an employee of the ACME Univesity. Alice has a profile.people record on OIP and so ACME University has a profile.institution.university.
How do we tell the world that Alice works for ACME-U? Simple, ACME-U must publish a record of type relationship.employment saying so.
In order to be thorough, we also built those record types.
Record type relationship.employment
Although this record type has the exact same fields as the record type relationship, we need to specify two roles: employer and employee.
These roles are records of the type profile.role that we built as an exact copy of profile. It simply has a name and a description of what it means.
Let's give an example.
ACME-U wants to tell the world that Alice is now an employee. The first need to explain what they mean by "employer" and by "employee".
First, they publish two records of the type role: one with the name "employee" and a description "Performs work for someone and get paid for it". Then another one with the name "employer" and description "Tell employees what to do and pay them for the work they have done".
Now, ACME-U makes a relationship.employment record wherein the field parties we have [ACME-U, Alice] and in field roles we have [employer, employee]. Now obviously, that OIP will put the transaction ids of the records of ACME-U, Alice, employer, and employee in place before it publishes.
Ok, now that ACME-U publishes the employment, it is like a job offer. To accept the offer, Alice must add the transaction id of that record to the fields validRelationships in her profile.
Now we all know that ACME-U offered a job to Alice and she took it. If they decide to terminate that relationship, ACME-U makes the record of employment with Alice inactive and Alice to accept the termination takes it out form her validRelationships
The same needs to happen between Alice's lab and Bob so the world knows that Bob is a member of Alice's lab. Notice that we mentioned Alice's lab, a record of type laboratory not Alice herself, a record of type people.
In the record type profile.institution.laboratory Alice is listed as a PI. Obviously, she has full control of her lab and these things could be interchangeable. In OIP it is not and we will learn about relationship.permission and its child types later when we approach OIP record ownership. By then we will see how ACME-U can make a record inactive and also all the other things that can be done to an existing record by the owner or by the profile holding the permission to do so.
Making changes in record types
How will it propagate if the parent uses field numbers as already in chain child types without breaking everything?
Updates of records in OIPd are done using version control, record schemas are records in the chain and their updates are also done using version control.
A change in a parent type won't break the existing child types because it will be a new version and the current child types are still compatible with the older version.
When searching for information, OIPd reads all the records built with all versions of the record type.
So... no breaks.
Record ownership
- talk about the confusion between private keys and ownership of OIP.
- How this is related to permission given to profile but in fact to the private key that owns that record.
Record manipulation
deactivate
edit
merge
- must own all records during merge
publish
split
- if we are doing the spliting on behalfOf someone else, who owns the splitted records?
transfer
Permissions
- Special type of relationship: introducing
reservedkeyword of protobuff.
Considerations on use plain text and referencing another record
- Plain text to be index with ES. Referencing will slow down JOIN searches to be done in the application side.
...
Concerns about publish a new record type
- There should be a field to explain what the record type tries to accomplish.