[Expo-tech] databaseReset.py

[Philip Sargent (Gmail)]

I have just pushed a somewhat radical change to databaseReset.py and I hope
no one gets too upset. 

 

It does the requested import to a scratch in-memory sqlite database (5.6
mins for the entire import) and only then does it do the import you
requested.

 

Since most of the runs of databaseReset are by people debugging code or
finding errors in input data this should save time overall quite a bit as
the errors are found during the scratch in-memory phase.


The reason for doing this is because I am looking forward to a long period
of fixing python 2to3 errors which will undoubtedly mean running all the
import code hundreds of times.



..and if I wrote the planned extension of skipping the full python file
import to the db and squirting the SQL generated by the in-memory phase into
the db instead - in a single transaction - the entire operation would be
much quicker under all circumstances.

 

Philip

 

PS Working on the import and the parsers is my way of learning how the db is
constructed out of the input data files. 

 

I think I have determined that the "surveyimgs" option : 

parsers.surveys.parseSurveys(logfile=settings.LOGFILE) 

does nothing functional at all and that this code is not in active use but
kept around in case any bits might be useful. 

 

The survey scan image files actually used in troggle are imported by
parsers.surveys.LoadListScans().

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wookware.org/pipermail/expo-tech/attachments/20200430/3d1f3536/attachment.htm>