[Expo-tech] QMs & svx2qm.py

Fri Apr 17 11:32:27 BST 2020

I added parsing the new survex QM style to the troggle survex parser on
expo last year and it looks like it got merged (though I'm not quite
sure how as it was part of some other parser work that still has a bug).
For troggle now the QMs have to fit this regex

r'^\s*QM(\d)\s+?([a-dA-DxX])\s+([\w\-]+)\.(\d+)\s+(([\w\-]+)\.(\d+)|\-)\s+(.+)$'

line 101 in the survex parser on master at the moment.

As I remember it the QMs get displayed as part of the cave page
somewhere if they exist but they usually don't as until I added this
only a few caves ever had QMs.

Sam

On 17/04/2020 01:19, Philip Sargent (Gmail) wrote:
> Actually it’s not as bad as that.
> 
> The QMs are all parsed when troggle imports the svx files. But Troggle
> just doesn’t have an **output page** to show them.
> 
>  
> 
> That should be a relatively easy addition.
> 
>  
> 
> *From:*Philip Sargent (Gmail) [mailto:philip.sargent at gmail.com]
> *Sent:* 16 April 2020 21:56
> *To:* 'Rob Watson'
> *Subject:* FW: QMs & svx2qm.py
> 
>  
> 
> Rob,
> 
> since you did the documentation for QMs, you might be interested in
> quite how broken the website/troggle system is for QMs.
> 
> We have a moderately clear architecture, but all the input procedures
> and parsers are all over the place.
> 
>  
> 
> At the moment it looks like the command line tool *svx2qm.py *does a
> good job in extracting all the QMs from all the svx files. So I will
> document that now.
> 
> This is completely independent of the website.
> 
>  
> 
> By next serious expo in 2021 I hope this part of the system is re-written.
> 
>  
> 
> Philip**
> 
>  
> 
>  
> 
> *From:*Philip Sargent (Gmail) [mailto:philip.sargent at gmail.com]
> *Sent:* 16 April 2020 21:43
> *To:* philip at tecnocode.co.uk; expo-tech at lists.wookware.org
> *Subject:* QMs & svx2qm.py
> 
>  
> 
> It looks like the files *svx2qm.py* doesn’t produce the files troggle
> imports, but some perl scripts do.
> 
> Which is even worse.
> 
>  
> 
> So it looks to me that Phil Withnall’s *svx2qm.py*code should be
> repurposed into a “proper” qm-import module which
> 
> would eventually live in troggle/parsers/
> 
>  
> 
> The perl and CSV stuff are in :expoweb: not :loser: repos.
> 
>  
> 
> troggle/parsers/qms.py currently imports into troggle using a mixture of
> csv and html parsers:
> 
>  
> 
> parseCaveQMs(cave='stein',inputFile=r"1623/204/qm.csv")
> 
> parseCaveQMs(cave='hauch',inputFile=r"1623/234/qm.csv")
> 
> parseCaveQMs(cave='kh', inputFile="1623/161/qmtodo.htm")
> 
> #parseCaveQMs(cave='balkonhoehle',inputFile=r"1623/264/qm.csv")
> 
>  
> 
> And possibly these .csv files are produced by
> 
> qmreader.pl
> 
> or possibly
> 
> tablize-qms.pl
> 
> which are all in expoweb/1623/204/.
> 
>  
> 
> 1623/204/qms.html is definitely produced by
> 
> tablize-qms.pl
> 
> from
> 
> qms.csv
> 
>  
> 
> in 1623/234
> 
> there is just tablize-qms.pl , qm.csv and qm.html
> 
>  
> 
> in 1623/161 there is the qmtodo.htm and a qmdone.htm but no csv file
> 
>  
> 
> in 1523/264 there is a qm.csv which apparently was hand-edited?
> 
>  
> 
> It all needs someone to properly take it in hand and sort it all out.
> 
>  
> 
> Note that this all predates Tunnocks; and 'balkonhoehle' is commented
> out. Nothing at all for FGH, GSM, HC etc.
> 
> Philip
> 
>  
> 
> *From:*Philip Withnall [mailto:philip at tecnocode.co.uk]
> *Sent:* 13 April 2020 23:41
> *To:* Philip Sargent (Gmail)
> *Subject:* Re: svx2qm
> 
>  
> 
> Hi Philip,
> 
>  
> 
> Hope you’re well, thanks for getting in touch about this.
> 
>  
> 
> On Sat, 2020-04-11 at 20:28 +0100, Philip Sargent (Gmail) wrote:
> 
>     Hi,
> 
>     I am trying to document the non-troggle scripts used by the expo
>     website.
> 
>     http://expo.survex.com/handbook/troggle/otherscripts.html
> 
>      
> 
>     I see that your *svx2qm.py* lives in the :loser: repo in /qms/
> 
>     But that means that the files generated (csv or human) are not
>     served by the webserver.
> 
>      
> 
>  
> 
> The generated files are not meant to be served by the webserver, it’s a
> tool for people to run locally. Someone could modify it to create HTML
> output (or post-process the CSV output to do the same), but that is work
> still to be done.
> 
>  
> 
>     I can't see any problem with moving it all to expoweb/scripts/ - so
>     long as it is run with the loser top level directory specified - but
>     I might be mistaken:
> 
>     find  /home/expo/loser -name '*.svx' | xargs ./svx2qm.py --format human
> 
>     and it should go into the Makefile too at some point.
> 
>      
> 
>  
> 
> Feel free to move it wherever; I am not planning on doing any further
> work on it. The script itself just expects to be passed some (relative
> or absolute) paths to SVX files, so can be placed wherever, as long as
> it’s passed appropriate relative paths.
> 
>  
> 
>     *How and who usually uses the output?* Is there another script
>     somewhere producing an HTML page ?
> 
>     Or should I do something like this
> 
>     http://expo.survex.com/xmlvalid.html
> 
>     or
> 
>     http://expo.survex.com/svxvalid.html   
> 
>     to display the text output ?
> 
>      
> 
>     Or maybe even edit it into a troggle "view" so that it become an
>     integral part of troggle ?
> 
>  
> 
> I haven’t written any other scripts which post-process the data or
> otherwise format it.
> 
>  
> 
> I guess it all depends on what questions people are trying to answer
> using the QM data, as to how (and where) best to present it. I’m afraid
> I don’t have any suggestions there.
> 
>  
> 
>     Rob Watson wrote some documentation about QMs
> 
>     http://expo.survex.com/handbook/survey/qmentry.html
> 
>     is there anything subtle missing  as to how they are used ?
> 
>      
> 
>  
> 
> Nope, I think Rob’s page covers it all. That page also documents the
> correct QM format which is what svx2qm.py understands. (There were some
> older or artisanal QM formats floating around at one point, although I
> think I reformatted them all so the tool would understand them, and so
> people would hopefully standardise on what Rob’s documented from then on.)
> 
>  
> 
> Philip
> 
> 
> _______________________________________________
> Expo-tech mailing list
> Expo-tech at lists.wookware.org
> https://lists.wookware.org/cgi-bin/mailman/listinfo/expo-tech
> 