[Expo-tech] Another modest proposal: core, minimal data model

Thu May 14 22:52:13 BST 2020

*Non-django troggle*
http://expo.survex.com/handbook/troggle/trogdesignx.html

Another possibility is ripping django out of troggle and leaving bare python plus a SQL database. This means that programmers would need to understand more SQL but would not need to understand "django". Arguably this could mean that we could gain. More people know some SQL than know django.

Writing our own multi-user code would not be sensible, hence the SQL database. But we could move to a read-only system where the only writing happens on data-import. Then we could use python 'pickle()' or 'json()' read-only data structures, but we would need to create all our own indexing and cross-referencing code.

There would be more lower-level code, but the different segments of the system could be in caving-sensible modules not django-meaningful modules. And we would not have all the extra language-like constructs that django introduces e.g. X.objects.set_all(), which modern editors complain about because it is a django idiom and not a function within the python codebase. (We could retain an HTML templating engine though.)

Caving-sensible modules *might* make updating easier, but that might be an illusion and software system modules
might turn out to be more separable.

+ see below

-----Original Message-----
From: Philip Sargent (Gmail) [mailto:philip.sargent at gmail.com] 
Sent: 13 May 2020 13:44
To: 'Wookey'
Subject: core, minimal data model

I think our core, minimal data model is actually quite intricate even if we 
exclude all the actual survey leg data and odd little things like "roles" 
for people in survex files. That needs documenting before we start on 
rewrites.

And troggle doesn't get it right always I think - there may be some 1:1 
mappings which are really m:n. But it is hard to disentangle the data model 
from reading the code as django can create automatic inverse mappings which 
means that the declarations are not as limited as they look.

I have a DB viewer which I use to browse the sqlite db. It is very helpful. 
A number of tables are completely unused.

Troggle particularly doesn't understand ARGE. All survex blocks are attached 
to a CUCC Expedition.

And troggle doesn't do QMs well at all. Sam was in the middle of trying to 
improve that.

Also, as you said, a Survex API would be a Good Thing. Or a greatly-enlarged 
set of command-line arguments to cavern.

New idea: forget python and django and databases, create a new 
*domain-specific language* specifically for doing this stuff.
Then we have a clear break between people changing stuff and the separate 
team who maintain the interpreter.

I am coming more and more to your & Rad's opinion. A dataset of json objects 
would be fine (though we would lose the diagnostic possibilities of a DB 
browser). We could initially use a standalone HTML template engine such as 
Jinja (based on django) https://wiki.python.org/moin/Templating for output. 
Though since the data very rarely changes we should cache every page after 
first view.

The parsers would be largely unchanged. We would want something like 
"flatpages" to enforce <head> CSS </head>.

[...]