Tutorial: processing SafeGraph in R with Apache Arrow

[Christian_Gunning_University_of_Georgia]

For errors with wide input files, see also the block_size parameter :

Error with Arrow package to read in multiple SafeGraph csv.gz files

I hit the same issue, seems like long fields (i.e. geometries, etc.) need more block space than the default settings. You can adjust the block_size parameter inline: dat <- open_dataset(data_path, format='csv', partitioning = c('month'), block_size=1e9) 1e9 did the trick for me, though for good practice you’d probably want to play around a bit with that.