Working With gDS Tables
The test platforms referenced in this website all use a Python-specific “global data store” called “gDS”.
See the README below for documentation on gDS V2:
https://talborough.github.io/tcsDocs/README_gDS/
​
gDS is unconvential. It's being presented because it works very well in test platforms. You can review its primary characteristics below. You can also use an alternative to gDS if you so choose.
​
See Case Study 1 for a tutorial on gDS.
How Are Tables Constructed in gDS?
gDS uses Python “list” and “dictionary” variables made global (located in shared memory) by the Python “multiprocessing” library. These variables are used just like any other Python variable. except that their values are seen by all the threads and processes created by the main Python code as it executes. It can be considered connectionless inter-process communication.
The Python “list” and “dictionary” variables can be combined into relationally organized tables and indices (in 3'rd-normal form if desired).
Together with the naming conventions used in the tables and indices are navigated and dereferenced cleanly and easily with ordinary Python code.
How Are Tables Constructed in gDS?
Creating "ordinary", "non-global" tables
Tables in gDS are built from collections of Python “lists.” Each Python list is a column of data in the table. For example:
​
gPerson_Name = []
gPerson_DOB = []
gPerson_SSN = []
gPerson_RowStatus = []
Python practitioners can see the above code contains no “special” Python syntax. The variables are not special in any way — it's just a plain-vanilla declaration of 4 ordinary lists. But, the table/lists are not in shared memory.
Creating Tables in Shared Memory
In order to make lists / tables in shared memory one does the following instead of the above:
from multiprocessing import Manager
gDSMgr = Manager()
gPerson_Name = gDSMgr.list()
gPerson_DOB = gDSMgr.list()
gPerson_SSN = gDSMgr.list()
gPerson_RowStatus = gDSMgr.list()
The code above creates four list variables in shared memory. The variables do not need to be cited in a Python “global” statement to be used anywhere in the running program or its spawned threads and processes.
Working with data in shared memory tables
o add one “row” of data to the above table/set of lists, use the Python “append” method just like a regular list:
gPerson_Name.append("Tom")
gPerson_DOB.append("1/1/1970")
gPerson_SSN.append("100-20-0300")
gPerson_RowStatus.append(None)
The above table is considered to contain one row of data, and that data is said to reside at “offset zero.” To print one column of that one new row of data:
print (gPerson_SSN[0])
To print out all the data in the table above, one would write in Python:
for personRef in range(len(gPerson_RowStatus)):
print (gPerson_Name[personRef],
gPerson_DOB[personRef],
gPerson_SSN[personRef])
Inter-Table References
A reference to a row in a table is an integer. If that integer is in another table you have a reference between 2 tables. The naming convention used for the referencing column is:
​
gAnimal_gFarm_Ref
which is a column in the Animal table that points to a row in the Farm table the animal belongs to. It can be dereferenced with Python code as:
​​
print (gFarm_Name[gAnimal_gFarm_Ref[animalRef]])
​​
where animalRef is a reference to a particular animal row and the farm name gets printed out.
​
gDS has a scheme for maintaining referential integrity in the face of row deletions.
gDS Compiler
A compiler, "gDSCompile", converts a gDS “.dd” (schema) file provided by the developer into Python code to define and help manipulate the tables defined by it. The “.py” code produced by gDSCompile is consumed by the test platform Python code when it starts up. The .py code can be reviewed separately to provide insight.
​​​​
Schema compiler - process a .dd: file and output an include file which:
​​​
Defines all the tables in the schema.
Defines helper/API routines per table:
<row reference> = <table>_AddARow(<column values>)
<table>_DumpRows(<row selection criteria>)
<reference adjustment list (RAL)> = <table>_DeleteRows(<row selection criteria>)
<table>_ApplyRALTo_<table>(<reference adjustment list>, <delete action>)
<table>_WriteToFile(<filespec>)
<table>_ReadFromFile(<filespec>)
​
For more documentation:
​
​Replace Transactions With Simpler Functionality
A traditional DBMS uses a "transaction" to manage concurrent access to the data in the tables. gDS offers alternatives to managing concurrency:
-
A single "system lock" is used to provide atomic operations on the data in the tables.
​​
-
Data can be organized so that each thread has it's own row of data.
​
-
Individual rows can be locked for use by a single thread.
​
-
Table data is memory-resident; loading data into it at the start does not take that long. Small amounts of "startup data" can be persisted if needed.
These alternatives are not embedded in gDS but used in the general flow of control in the test platform.