[Expo-tech] QMs & svx2qm.py

Fri Apr 17 01:19:00 BST 2020

Actually it’s not as bad as that.

The QMs are all parsed when troggle imports the svx files. But Troggle just doesn’t have an *output page* to show them. 

That should be a relatively easy addition.

From: Philip Sargent (Gmail) [mailto:philip.sargent at gmail.com] 
Sent: 16 April 2020 21:56
To: 'Rob Watson'
Subject: FW: QMs & svx2qm.py

Rob,

since you did the documentation for QMs, you might be interested in quite how broken the website/troggle system is for QMs.

We have a moderately clear architecture, but all the input procedures and parsers are all over the place.

At the moment it looks like the command line tool svx2qm.py does a good job in extracting all the QMs from all the svx files. So I will document that now.

This is completely independent of the website.

By next serious expo in 2021 I hope this part of the system is re-written.

Philip

From: Philip Sargent (Gmail) [mailto:philip.sargent at gmail.com] 
Sent: 16 April 2020 21:43
To: philip at tecnocode.co.uk; expo-tech at lists.wookware.org
Subject: QMs & svx2qm.py

It looks like the files svx2qm.py doesn’t produce the files troggle imports, but some perl scripts do.

Which is even worse.

So it looks to me that Phil Withnall’s svx2qm.py code should be repurposed into a “proper” qm-import module which

would eventually live in troggle/parsers/ 

The perl and CSV stuff are in :expoweb: not :loser: repos.

troggle/parsers/qms.py currently imports into troggle using a mixture of csv and html parsers:

parseCaveQMs(cave='stein',inputFile=r"1623/204/qm.csv")

parseCaveQMs(cave='hauch',inputFile=r"1623/234/qm.csv")

parseCaveQMs(cave='kh', inputFile="1623/161/qmtodo.htm")

#parseCaveQMs(cave='balkonhoehle',inputFile=r"1623/264/qm.csv")

And possibly these .csv files are produced by 

qmreader.pl

or possibly

tablize-qms.pl

which are all in expoweb/1623/204/.

1623/204/qms.html is definitely produced by

tablize-qms.pl

from

qms.csv

in 1623/234

there is just tablize-qms.pl , qm.csv and qm.html

in 1623/161 there is the qmtodo.htm and a qmdone.htm but no csv file

in 1523/264 there is a qm.csv which apparently was hand-edited?

It all needs someone to properly take it in hand and sort it all out.

Note that this all predates Tunnocks; and 'balkonhoehle' is commented out. Nothing at all for FGH, GSM, HC etc.

Philip

From: Philip Withnall [mailto:philip at tecnocode.co.uk] 
Sent: 13 April 2020 23:41
To: Philip Sargent (Gmail)
Subject: Re: svx2qm

Hi Philip,

Hope you’re well, thanks for getting in touch about this.

On Sat, 2020-04-11 at 20:28 +0100, Philip Sargent (Gmail) wrote:

Hi,

I am trying to document the non-troggle scripts used by the expo website.

http://expo.survex.com/handbook/troggle/otherscripts.html 

I see that your svx2qm.py lives in the :loser: repo in /qms/

But that means that the files generated (csv or human) are not served by the webserver.

The generated files are not meant to be served by the webserver, it’s a tool for people to run locally. Someone could modify it to create HTML output (or post-process the CSV output to do the same), but that is work still to be done.

I can't see any problem with moving it all to expoweb/scripts/ - so long as it is run with the loser top level directory specified - but I might be mistaken:

find  /home/expo/loser -name '*.svx' | xargs ./svx2qm.py --format human

and it should go into the Makefile too at some point.

Feel free to move it wherever; I am not planning on doing any further work on it. The script itself just expects to be passed some (relative or absolute) paths to SVX files, so can be placed wherever, as long as it’s passed appropriate relative paths.

How and who usually uses the output? Is there another script somewhere producing an HTML page ?

Or should I do something like this

http://expo.survex.com/xmlvalid.html 

or

http://expo.survex.com/svxvalid.html   

to display the text output ?

Or maybe even edit it into a troggle "view" so that it become an integral part of troggle ?

I haven’t written any other scripts which post-process the data or otherwise format it.

I guess it all depends on what questions people are trying to answer using the QM data, as to how (and where) best to present it. I’m afraid I don’t have any suggestions there.

Rob Watson wrote some documentation about QMs

http://expo.survex.com/handbook/survey/qmentry.html 

is there anything subtle missing  as to how they are used ?

Nope, I think Rob’s page covers it all. That page also documents the correct QM format which is what svx2qm.py understands. (There were some older or artisanal QM formats floating around at one point, although I think I reformatted them all so the tool would understand them, and so people would hopefully standardise on what Rob’s documented from then on.)

Philip

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wookware.org/pipermail/expo-tech/attachments/20200417/e3e7d2d1/attachment-0001.htm>