[Expo-tech] troggle import speedup again, now < 11 mins

[Philip Sargent (Gmail)]

Previously I had turned off the database storage of all the SurvexLegs as they were never used (there was no code reading them).

 

I have now managed to get the survex parser to not store any of the 36,000 SurvexStations either,  but about 350 of these were needed for the prospecting guide so I have imported/created them separately using the survexpos information.

 

The entire import now takes 630s on my PC: 10 and a half minutes (to disc SSD).

It is still parsing (but not storing) everything in the survex files and this takes nearly half the time.

The database (sqlite) is now 5,452kB or 56% of its previous size.

 

In doing this I tripped over a dozen labelling mismatches between the SurvexBlock data from survex files and the supposedly-matching cave entrance labels. These now get reported as DataIssues in expo://admin/core/dataissue/

e.g.  <http://127.0.0.1:3000/admin/core/dataissue/21/> .1623.2004-18to2006-09      p2006-09

 

Tomorrow I’ll back-port this to the python2.7 version and upload to the server.

Now that’s done, everything else should go a lot more quickly.

 

I have a lot of de-crufting to do now that I can see the wood for the trees.

 

 

On the django version front, I have at last succeeded with bell, book and candle in exorcising the fossil remains from the depths of django’s impenetrable ‘migrations’ memory. So troggle (python3.7.7) is “fully” working* with just:

 

Django 1.7.4

Pillow 7.1

setuptools 41.2.0

Unidecode 1.1.0

 

having removed all these:

django-extensions

django-registration

django-tinymce

six 1.14

 

and also the Django internal ‘apps’:

'django.contrib.sites'

'django.contrib.redirects'

 

The secret is to never remove the package from the OS (or venv) until django has been through several rebuilds with all its internal references removed and the database scrubbed with Brillo pads.

 

I will add back django-extensions once I have got django as far up to recent release levels as is convenient. Unless I need some of its tools to disentangle more Migrations issues. 

I have given up on creating a clean Debian machine as a mimic for the server. I’d already had enough of ‘export Display=:0.0’ 30 years ago thanks.

 

Philip

 

* i.e. working about as well as it was before. Though 19 people with complex surnames may disagree.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wookware.org/pipermail/expo-tech/attachments/20200615/526bad6c/attachment.htm>