FEATURE: Ability to upload JSON files larger than 100MB

[mingfang]

I'm trying to import this file https://data.opensanctions.org/datasets/20231205/peps/entities.ftm.json
but get the error

Traceback (most recent call last):                                                                                                                                                                                                                   
  File "/ingestors/ingestors/manager.py", line 220, in ingest                                                                                                                                                                                        
    self.delegate(ingestor_class, file_path, entity)                                                                                                                                                                                                 
  File "/ingestors/ingestors/manager.py", line 245, in delegate                                                                                                                                                                                      
    ingestor_class(self).ingest(file_path, entity)                                                                                                                                                                                                   
  File "/ingestors/ingestors/misc/jsonfile.py", line 32, in ingest                                                                                                                                                                                   
    raise ProcessingException("JSON file is too large.")                                                                                                                                                                                             
ingestors.exc.ProcessingException: JSON file is too large.      

https://github.com/alephdata/ingest-file/blob/3860024df29dc04d5f37dd47aee85f936f93c040/ingestors/misc/jsonfile.py#L16

[pudo]

Since the file in question is in Aleph format, you can also use the alephclient write-entities -i entities.ftm.json command to import the contained entities in chunks, without making the whole file go through an ingest process.