Serialise sf as plain lists?

[MilesMcBain]

The decode speeds of wkb are a pain on large datasets. Could we just serialise sfc as plain lists? Sacrificing some size for speed?

[MilesMcBain]

@mdsumner has some opinions!

So some background here:

We piggyback a fair bit of stuff off {targets} ability to use AWS s3 as a store for targets. So we have plans that write their targets up in the cloud, and then we pull them down in other pipelines, shiny apps etc.

For a while, this was broken because {targets} didn't let you choose custom serialisation for your targets going to s3. There was a limited menu of choices. This may have changed, not sure. Anyway it supported parquet via arrow, which was great for plain rectangles, but basically unusably slow for {sf}. This was mainly to do with the crappy performance of arrow on list columns, which as has been improved a bit also.

The solution here was the best we could come up with quickly: targets can just use plain old write_parquet, which under the hood, now detects and serialises sfc to WKB. On the flipside, read_parquet now detects spatial metadata and blob columns and deserialises from WKB to sfc.

[MilesMcBain]

As I said in our twitter convo: The nice thing about this is that reading a target from an s3 cache works like normal in that you don't have to know the type of the object to get it out of the cache into your R session. You just need to know the name, and I think this is a desirable property we want to keep.

[mdsumner]

cool ... fwiw, I wouldn't have understood that targets seralization need until a couple weeks ago ...

just in answer to your OP - which is probably not of interest now - can roundtrip sf very fast with sfheaders - you need to convert to df from sf or sfc (just a data frame of x, y, [linestring_id, polygon_id, multipolygon_id] (options appropriate to the geometry type) - need handling for what is the return trip geometry type but some automation with sfheaders::sf_mpoly and friends

in terms of WKB, you can convert fast from and to sf with {wk} (or some combination of Dewey's stuff I'm not completely across where that's at atm), but very happy to get involved with specifics

finally, just a spruik for {vapour} in case that fits anywhere, you can read fields or geometry independently via GDAL - get raw WKB (no need for conversion to in memory R for example) - if you only need the fields from a layer

I'll explore what's happening here and see what might be faster with {wk} and friends

[MilesMcBain]

The {sfheaders} idea is a good one... so we could flatten geometry out into more columns and store some metadata that says how to recombine them.

I'll check out {wk} too thanks. Maybe that offers the best drop-in option for what's happening here.

Not sure if vapour fits in. Our workflow is usually either:

• pull geometry from database -> use in R -> write to s3 parquet (as our psuedo-sf thing)
• pull psuedo-sf thing from s3 parquet-> use in R as sf -> write to s3 parquet (as our psuedo-sf thing).

[mdsumner]

I do think wk will rip for what you're doing - but, I've lost track of the best steps, one of Dewey's blog posts will sort

what is the "pull geometry from database" step like? is that with read_sf(), or more generic DBI/dbplyr? (if latter, you get a geometry column sometimes? )