[Expo-tech] Memory

[Wookey]

On 2019-07-11 00:57 +0100, Sam Wenham wrote:
> Crap
> 
> Loading Pos....
> Traceback (most recent call last):
>   File "databaseReset.py", line 206, in <module>
>     reset()
>   File "databaseReset.py", line 93, in reset
>     import_survex()
>   File "databaseReset.py", line 66, in import_survex
>     parsers.survex.LoadPos()
>   File "/home/expo/repositories/git/troggle/parsers/survex.py", line
> 284, in LoadPos
>     call([settings.CAVERN, "--output=%s%s.3d" % (settings.SURVEX_DATA,
> settings.SURVEX_TOPNAME), "%s%s.svx" % (settings.SURVEX_DATA,
> settings.SURVEX_TOPNAME)])
>   File "/usr/lib/python2.7/subprocess.py", line 168, in call
>     return Popen(*popenargs, **kwargs).wait()
>   File "/usr/lib/python2.7/subprocess.py", line 390, in __init__
>     errread, errwrite)
>   File "/usr/lib/python2.7/subprocess.py", line 916, in _execute_child
>     self.pid = os.fork()
> OSError: [Errno 12] Cannot allocate memory
> 
> In the process of a DB reset (full)

Hmm. It worked before. As it's the middle of the night I've stopped
apache (using 8% of memory) and am trying again. Bit unfortunate if we
have to stop apache to reliably databasereset. But there seems to be
plenty of memory otherwise so I wonder if it can be adjusted to use
less RAM.

I started off with 130M used, mostly by mysql.
mysql rises from 8% to 10.3% (104MB), then slowly creeps up to 11.5% (114M) 
databaserest uses 8.3% so we are up to 228M (for the entrances and caves sections)
gets up to 10% (250M) doing surveyscans
rises steadily doing GetPersonExpeditionNameLookups for each year and parsing kataster/../caves-*  
260 Mb, 267MB, 274MB, 292MB (14%)..., 
300 (15%, andh mysql up to 10.9% (108MB), 
321MB (170MB dbr res) 17%, and 11% mysql (109MB res), 
342MB (190MB dbr res) 19%, 
352MB (199MB dbr res) 20%, 372 (219) 22%, 
400MB (246MB dbr res) 24.7% (111MB of mysql), 
445MB (289MB dbr res) 29% ( to do 204), 
475MB (318MB dbr res) 32% (to do 258), 
536MB (377MB dbr res) 37.9% (to do 264), 
557MB (400MB dbr res) 40.2% % (to do surface), 
673MB (517MB dbr res) 51.8% (to do gps/gpx) (20MB to do one 200K gpx file), 
then starts doing 
('loading', <Cave: 1623-foo>, u'1623')
(u'foo,'u'1623'), and re-reading all the files (as caves-1623/264, i.e not under kataster../caves-1623/ as it did above)
mysqld falls to 95MB res, 9.4% 
734MB (572MB dbr res) 57.4%
762MB (605MB dbr res) 60.9% (loading 258)
774MB (633MB dbr res) 63.6%
787MB (648MB drv res) 65.1%

so stopping it reading everything in twice and parsing all the gpx
data would reduce footprint by ~400MB, and take an hour off the runtime. 
Not sure if this is new or it's always worked this way. This seems to be 
LoadAllSurvexBlocks() in survex.py, and even has a fixme: 
 #Load each cave, 
 #FIXME this should be dealt with load all above
I'm not quite sure what's going on there, but it's clearly epically 
inefficient and memory-intensive. Not surprised it runs out of memory!

I've left it running, and maybe it'll finish?
someone needs to restart apache if it's finished (or you want the
webserver back anyway). 

The survex dataset parsing takes over 2 hours as well as using over
500MB RAM - I wonder if it can be made more efficient in either
regard.

On the way I noticed it whinging about 'tape misread' errors in
'neartc' and 'p78'.

Wookey
-- 
Principal hats:  Linaro, Debian, Wookware, ARM
http://wookware.org/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.wookware.org/pipermail/expo-tech/attachments/20190711/7f175104/attachment.sig>