Giters

NetTopologySuite / NetTopologySuite.IO.GeoJSON

GeoJSON IO module for NTS.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Can't modify STJ's Attribute Table

HarelM opened this issue · comments

commented

I'm getting the following error when trying to add a field to an AttributeTable after deserializing a feature using the STJ
NotSupportedException: Modifying this attribute table is not supported.
Is this a missing feature?
Here's the failing test if that helps:

        [TestMethod]
        public void AddAttributeFailure()
        {
            string str = @"
                 {
                    ""type"": ""Feature"",
                    ""geometry"": {
                        ""type"": ""Point"",
                        ""coordinates"" : [
                            35.0666687,
                            32.5499986
                        ]
                    },
                    ""properties"": {
                        ""accuracy"": ""minutes""
                    },
                    ""poiId"": ""OSM_node_278470424"",
                    ""poiGeolocation"": {
                        ""lat"": 32.5499986,
                        ""lon"": 35.0666687
                    },
                    ""poiAlt"": 172.0123568908041
                }";
            var factory = new GeoJsonConverterFactory();
            var indentedOptions = new JsonSerializerOptions
            {
                DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
                WriteIndented = true
            };
            indentedOptions.Converters.Add(factory);

            var f = JsonSerializer.Deserialize<IFeature>(str, indentedOptions);
            Assert.IsNotNull(f);
            f.Attributes.Add("test", "test");
            Assert.IsTrue(f.Attributes.Exists("test"));
        }
commented

4STJ attributes tables are read-only, as System.Text.Json didn't have support for writable DOM objects until version 6.

Now that it has this functionality, we could look into bumping up our minimum supported version of that dependency. EDIT: turns out that we're already demanding 6.0.3 or higher, so there's not even a need to bump that up! How fortunate.

commented

I honestly can't find documentation about how to edit/modify/clone-update a JsonElement which is the underlying data structure below JsonElementAttributeTable.
Having said that, I'm not sure I fully understand the decision to use that as an underlaying data structure.
Selecting that basically breaks the API and IAttributeTable interface as there are "unimplemented" methods.
I'll be happy to help push this forward as this is currently a blocker for me to finish the STJ migration.
Yes, I know, I can make a "convert to regular AttributeTable" and then use that, but it feels a bit awkward...

commented

I honestly can't find documentation about how to edit/modify/clone-update a JsonElement which is the underlying data structure below JsonElementAttributeTable.

I'm working on it right now. In case I disappear before finishing it, the important part is: https://learn.microsoft.com/en-us/dotnet/api/system.text.json.nodes.jsonobject.create

Selecting that basically breaks the API and IAttributeTable interface as there are "unimplemented" methods.

Sort-of... I'd say that it's a bit more like ICollection<T> when IsReadOnly is false. I know that we don't have an IsReadOnly that you can use to check this in advance, so it's admittedly worse here than in ICollection<T>.

Yes, I know, I can make a "convert to regular AttributeTable" and then use that

That's the reasoning behind the decision. The intentional design limitations of System.Text.Json prevent us from deserializing an entire object graph of any arbitrary top-level type ahead-of-time using only information that's present in the GeoJSON object, so we had to look at other ways of representing the nested structure.

There are a number of viable options here, each with their own pros and cons. The approach that we took allows the system to do the minimum amount of work that's necessary to support reading the values, which I imagine is all that the overwhelming majority of consumers need from it. And because it throws NotSupportedException when you try to write anything, we can always add that support later without breaking anything that was working before.

commented

So I'm guessing you are planning to switch from JsonElement to JsonObject, right?
I'm having a hard time opening and running this project in Rider for some reason, not sure why... :-/
In any case, if you have a pre-release nuget package for STJ you can share that allow modifying the attribute table without knowing the underlaying implementation let me know and I'll be happy to test it.
I'm guessing that adding support just for that shouldn't be too difficult...

commented

So I'm guessing you are planning to switch from JsonElement to JsonObject, right?

As nice as that sounds, it's nothing quite so simple, I'm afraid...

In nearly every case, I expect the overhead involved in creating a writable object is going to go to waste, so I don't want to do that eagerly. The whole reason why System.Text.Json exists (as I understand it) is because Newtonsoft.Json's design forces a significant amount of overhead onto every consumer, in order to support features that a majority of consumers don't need a majority of the time.

Throughout my involvement in the design and development of NTS.IO.GeoJSON4STJ, I've made it a personal goal to stay true to that design philosophy. That is consistent with the factors that I feel are most likely to motivate people to choose this library over NTS.IO.GeoJSON.

So to that end, my overall plan is as follows (this is subject to change if needed, of course!):

  1. For consumers who only ever need to read the attributes, everything will remain exactly the same as it's always been, and none of the rest of this list will apply to them.
    • Technically, they will likely have to live with the table object itself taking up an additional pointer-sized object reference and deal with some trivially more complicated logic at the start of most methods... all within the same ballpark as the overhead of JsonElement itself, though.
  2. The first time someone modifies an attributes table, we will copy the JsonElement to a JsonObject that we hold onto in order to support those modifications. Once that JsonObject exists, all the other methods will use it instead of the JsonElement that it was created from.
  3. Further modifications will edit that same JsonObject.
  4. After making any modifications, trying to externally access the RootElement property will do... something (haven't gotten this far yet in my plan). Reasonable-sounding options I can think of while typing this up are:
    • Return the original value that it was created from.
    • Return a value that represents the current state No further modifications will be propagated to this value.
    • (I don't think it's possible to return a value that will also observe further modifications. I haven't checked, though, so this bullet point is here to cover that.)
    • Throw an exception because it's unclear what alternative semantics make more sense, and this probably isn't all that important anyway, so defer the decision point until someone opens another issue like this one.

This is probably one of the more complicated ways to solve this problem... what I would have liked to have had is an IAttributesTable.IsReadOnly flag on that I could have pointed you to, along with a simple .ToWritable() extension method (or similar) that you could use to opt-in to all this overhead. But adding members to interfaces is always a breaking change (in .NET Standard 2.0, anyway...), and so if IsReadOnly wasn't there before, it's problematic to add it later.

In any case, if you have a pre-release nuget package for STJ you can share that allow modifying the attribute table without knowing the underlaying implementation let me know and I'll be happy to test it.
I'm guessing that adding support just for that shouldn't be too difficult...

I think I don't quite follow your proposal. The only reasonable cheap alternative I can think of immediately is to add some constructor overload(s) to AttributesTable that let you initialize it using the values from some other IAttributesTable object. Is that something like what you're suggesting?

commented

The last part assumes this isn't too difficult too implement (allowing modification to attribute table), so if you implement it and want me to test it I'll be happy to, using a pre-release nuget package I guess...

commented

The problem I'm running into when trying to implement the idea I mentioned in #117 (comment) is that a JSON object that's nested inside of a JsonElementAttributesTable is exposed as an inner JsonElementAttributesTable. If we're going to allow modifications to this type, then modifications to an instance should be reflected in all containing instances.

The more I dug into this, the more problems I found that System.Text.Json.Nodes just flat-out solves. So my new plan is to create a separate IAttributesTable implementation that's backed by a JsonObject, and add an opt-in flag to GeoJsonConverterFactory that tells us to use it when deserializing features.

commented

Cool! Thanks for looking into it!!
Let me know when you have a pre-release Nuget package I can take for a spin :-)

commented

Let me know when you have a pre-release Nuget package I can take for a spin :-)

Starting from now, 3.1.0 prerelease packages on https://www.myget.org/feed/nettopologysuite/package/nuget/NetTopologySuite.IO.GeoJSON4STJ include this change.

I was hoping not to resort to making it an opt-in affair, but now that JsonElementAttributesTable is publicly documented to be the type of our IAttributesTable instances, nested tables complicate the issue tremendously. I believe that it is likely important that edits to the nested tables should be visible in their containers, and getting that right would ultimately involve reimplementing quite a lot of how System.Text.Json.Nodes handles essentially the same problem.

  • For reasons that should be obvious in hindsight, System.Text.Json.Nodes pretty much does (a better job at) exactly what I was intending to do: everything is served directly from a JsonElement until you need to do something that can make changes, at which point all containers of the object being modified get opened up to accommodate.

So rather than duplicate all of this built-in code (but worse, probably), I went for the approach that was much simpler to implement and maintain, which was to keep this as just a thin wrapper around the built-in JSON DOM.

I've also opened NetTopologySuite/NetTopologySuite.Features#16 for looking into better ways of supporting attributes tables that shouldn't necessarily need to support being modified in-place.

Finally, in my defense, I should emphasize that although I did inject a lot of what I'll call "@airbreather isms" into the 4STJ design and implementation, including the part where I knowingly made it impossible to modify the IAttributesTable object (see #50), the entire reason why 4STJ attributes are handled so much differently than their counterparts in Newtonsoft.Json is because of the issues discussed on dotnet/runtime#30969.

  • Unlike in Newtonsoft.Json, System.Text.Json intentionally doesn't give us the facilities that we would need to be able to dynamically deserialize objects that are nested in GeoJSON Feature objects' properties into whatever arbitrary CLR type you would want (using only information that's present in the file, anyway).
  • Therefore, the way we chose to do it, we keep just enough of the JSON source around inside of the IAttributesTable so that the caller can take it the rest of the way to whatever more strongly-typed object they might need.
  • This wasn't the only possible way to do this. Another alternative could have been to create normal AttributesTable objects and make "convert something from IFeature.Attributes into my strongly-typed CLR object model" a second-class citizen that would have been a lot harder to get "just right".
  • As someone who was also coincidentally working on building out a brand new GeoJSON-centric ASP.NET Core WebAPI project at the time, I judged that letting people modify these objects in-place was much less important than streamlining the process of reading their nested objects using the same JSON-to-CLR converters that they used when reading the rest of their JSON documents, so I focused on something that would make this happen, without closing the door on supporting in-place edits at some point later in the future.
  • My mistake was in not giving enough weight to the fact that an IAttributesTable doesn't have a way for consumers to actually discover that a particular instance doesn't actually support in-place modification, apart from just trying it and seeing the error that you saw. Now, I'll defend my workflow of "someone tries something --> it throws an error that I put in because I'm too lazy to support that thing fully --> they complain to me --> I address the issue after they show me an important use case" to the ends of the earth for truly new development, but it's harder to defend here when it's so easy to "just" switch a pre-existing nontrivial application from the Newtonsoft.Json version to 4STJ.
commented

This looks like it is working as expected. Thanks!!
We all make mistakes :-) as long as we can correct them it fine. Good thing I chose to code and not to do brain surgery. :-P

commented

Let me know when a new official version is available so I can remove the temporary myget source.
THANKS!

commented

Let me know when a new official version is available so I can remove the temporary myget source.
THANKS!

@HarelM this is the notification that you requested.

commented

Cool! THANKS! Keep up the good work!