Revision 82dd582045b2ec890b46e78d5419389be4916146 (click the page title to view the current version)
Changes from beginning to 82dd582045b2ec890b46e78d5419389be4916146
---
format: markdown
toc: yes
title: Pygolem toolkit
categories: Analysis, SW
...
Pygolem is software developed for the tokamak GOLEM database. It provides easy options for loading and saving of data, plotting and advanced data processing. Pygolem is written in Python, however our implementation strongly depends on bash scripts and makefiles.
Use code previews in the GOLEM database for more examples
FAQ:
==========
Basic examples / clients for several programming languages
-------------------------------------------------------------------------------
**See more details in [DataProcessing ](/SW/DataProcessing) section** <br>
**Try our simple web interface** [[link]](http://golem.fjfi.cvut.cz/utils/golplot) [[examples]](http://golem.fjfi.cvut.cz/wiki/SW/ShowRoom/index)
Use pygolem web interface
------------------------------------------------------------------------------
If you need to download data from many shots, the simplest way is to use pygolem web interface. You have to know names of your diagnostics (see the <a href="/shots/0/Data.php">Data</a> page) and you can either use the links provided by the Data page or if you prefer another data formats use following path
**``http://golem.fjfi.cvut.cz/cgi-bin/data/12345/<your diagn name>_<options>.<format>``**
supported formats are
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field">
<th class="field-name">None </th><td class="field-body"> Ordinary TAB separated ASCII format</td>
</tr>
<tr class="field">
<th class="field-name">.txt </th><td class="field-body"> Ordinary TAB separated ASCII format</td>
</tr>
<tr class="field">
<th class="field-name">.gz</th><td class="field-body">Zip compressed ordinary TAB separated ASCII format</td>
</tr>
<tr class="field">
<th class="field-name">.csv</th><td class="field-body">Comma / semicolon separated ASCII values</td>
</tr>
<tr class="field">
<th class="field-name">.npy</th><td class="field-body"> Standard python binary data</td>
</tr>
<tr class="field">
<th class="field-name">.npz</th><td class="field-body"> Compressed python binary data (recommended)</td>
</tr>
<tr class="field">
<th class="field-name">.mat</th><td class="field-body">Matlab binary data</td>
</tr>
<tr class="field">
<th class="field-name">.xls</th><td class="field-body">Excel 97 format</td>
</tr>
<tr class="field">
<th class="field-name">image</th><td class="field-body">Supported formats: png, svg, svgz, pdf</td>
</tr>
</tr>
<tr class="field">
<th class="field-name">.link</th><td class="field-body"> return a http link to the stored data </td>
</tr>
<tr class="field">
<th class="field-name">.pro</th><td class="field-body">IDL - not supported, use .mat files (<a href="https://github.com/gordonfarquharson/idl-load-mat/blob/master/load_mat.pro">load_mat.pro</a>)</td>
</tr>
</tbody>
</table>
for the ASCII formats you can select if you prefer dots/commas decimal separators
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field">
<th class="field-name">_dp</th><td class="field-body"> Dots point separators, selected as default !! </td>
</tr>
<tr class="field">
<th class="field-name">_cp</th><td class="field-body">Comma points separator</td>
</tr>
</tbody>
</table>
**Examples:** <br>
[``http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage``](http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage) - returns simple ASCII data <br>
[``http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage.npz``](http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage.npz) - returns compressed loop voltage data <br>
[``http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage.csv``](http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage.csv)- comma separated csv - "czech excel" <br>
[``http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage.xls``](http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage.xls) - Excel/OO Calc/Gnumeric <br>
[``http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage.png``](http://golem.fjfi.cvut.cz/cgi-bin/data/0/loop_voltage.png) - A simple plot of the data <br>
Load data in your page
------------------------------
use code i.e for loop voltage
~~~ { .python }
from pygolem_lite import Shot
tvec, data = Shot()['loop_voltage']
~~~
and you will download data for shot number where is currently placed your script otherwise use
~~~ { .python }
from pygolem_lite import Shot
from matplotlib.pyplot import *
tvec, data = Shot(10322)['loop_voltage']
plot(tvec, data)
show()
~~~
if you want to download data from DAS "nistandard":
~~~ { .python }
tvec, data = Shot(10322)['nistandard']
~~~
if you need only one channel you can use either number or <a href="/shots/0/das_configuration.cfg"> channel id </a>
~~~ { .python }
tvec, data = Shot(10322)['nistandard', 0]
tvec, data = Shot(10322)['nistandard', 'uloop']
~~~
full list of accessible signal is in [golem.fjfi.cvut.cz/shots/0/Data.php]( http://golem.fjfi.cvut.cz/shots/0/Data.php ) <br>
Is is also possible to use wget and pygolem web interface. See more examples in section ``Advanced examples``.
Add data from your folder or new DAS to database
-----------------------------------------------------------------
All data settings can be done in
[/golem/database/velin/includes/data_configuration.cfg](/SW/pygolem/data_configuration.cfg)
and DAS is in
[/golem/database/velin/includes/das_configuration.cfg](/SW/pygolem/das_configuration.cfg)
Use the examples to add new diagnostics. Supported data formats are: **npy, npz, None, gz, txt, csv, lvm**, you do not need to add the file ending to the config.
Plot data using includes modules (keep the style)
------------------------------
Recommended way: find a page with similar plot and use copy&paste <br>
You can use ie. matplotlib, gnuplot, octave, ...
Write a python template
------------------------------
Read manual <a href="http://www.makotemplates.org/" > http://www.makotemplates.org/ </a>, or just copy&paste the code from existing pages - for example DAS or Plasma Position pages<br>
PS: use ## if you want to comment a code outside of code block "<% your code %>"
Sign if your page failed or not ( ✓, ✕, ?)
-------------------------------------------------------------------------
Write nonzero to status file in your folder if you failed
~~~ { .python }
saveconst('status', 0) => OK
saveconst('status', 1) => ERROR
saveconst('status', 2) => WORKING
saveconst('status', 3) => UNKNOWN
saveconst('status', 4) => NONE
~~~
or if the active code is in the python template
~~~ { .python }
page_path, base_path, page = get_page_paths(shot, page)
saveconst( page_path + '/status', 0)
~~~
<font color="red" size=3>`` Failure - ✕ - is expected as default `` </font>
Integrate your data to pygolem
-------------------------------------------------------
If you want to share your results with others, it is recommended to add your results to pygolem.
<br>Add your results to [/golem/database/velin/includes/data_configuration.cfg](/SW/pygolem/data_configuration.cfg) in include directory.
You can show accessible data in your .template page with the following code:
~~~ { .python }
Shot(shot).get_pygolem_list(['plasma_position_r', 'plasma_position_z', 'plasma_radius'])
~~~
Load basic constants / settings
-----------------------------------------
If you need to load basic calibration constants, use:
~~~ { .python }
from pygolem_lite import config
~~~
The config from the last shot is available here: <a href="/0/basicdiagn/config.py">config.py</a>
Advanced examples
==============
if you need more options, you can use more functions of the class [``Data``](/SW/pygolem/pygolem_lite.py) (derived from ndarray):
~~~ { .python }
import matplotlib
matplotlib.rcParams['backend'] = 'Agg' # plotting on background
matplotlib.rc('font', size='10')
matplotlib.rc('text', usetex=True) # latex font
from matplotlib.pyplot import *
from pygolem_lite import Shot
_, data = Shot(10322)['nistandard']
data = data[::100,0] # data are downsampled and only the first column is used
errorbar(data.tvec, data, xerr=data.tvec_err, yerr=data.data_err, label=data.labels) # plot data, use resampled time vector and data
xlabel(data.ax_labels[0]) # use default labels
ylabel(data.ax_labels[1])
title(data.info) # use diagnostic / DAS name
savefig( data.name+".png" ) # save using diagn. id
~~~
Another example for 2D plotting
~~~ { .python }
from matplotlib.pyplot import *
from pygolem_lite import Shot
data = Shot(10663)['nistandard'][1]
data = data[::1000,:] # data are downsampled and only the first column is used
data *= 1000 # data are rescaled
pcolor(data.axis[0], data.axis[1], data.T)
xlabel(data.ax_labels[0])
ylabel(data.ax_labels[1])
title(data.info) # use diagnostic / DAS name
savefig( data.name+".png" ) # save using diagn. id
~~~
for more plotting examples use [tutorial](http://matplotlib.org/users/pyplot_tutorial.html)
Use of integrated plotting functions
------------------------------------------------
Our framework contains interface for uniform standard plots that can be simply used. Plotting is done using two basic functions [get_data and multiplot](/SW/pygolem#modules.py). The simplest example:
~~~ { .python }
from pygolem_lite.modules import get_data, multiplot
obj = get_data('loop_voltage')
multiplot(obj, plot_title , file_name , figsize = (9,4) )
~~~
where get_data supports several options how to load data :
- input is name from pygolem database ie 'loop_voltage', then details are loaded from database
- input is path to file in a standard format (ascii matrix or preferably data saved using save_adv - see pygolem.py), note that more additional details should be added to get_data function -
- list containing [tvec, data]
- Data object
Outputs from get_data can be arranged in list so select order in subplots. For example [obj1, [obj2, obj3]] will create 2 subplots, the first contains obj1, the second obj2+obj3.
~~~ { .python }
from pygolem_lite.modules import get_data, multiplot
data = [[ get_data('electron_temperature', 'Electron temperature', 'T [eV]' , ylim = [0,None] , reduction = True),
get_data('spectrometr\temperature', '$T_e$ spectrometer', 'T [eV]' ,ylim = [0,None], fmt='r--') ],
get_data('plasma_current', 'Plasma current', 'I$_{p}$ [kA]' , ylim = [0,None], data_rescale = 1e-3, reduction = True) ]
multiplot(data, 'Nice graph', 'plot', figsize = (9,5) , file_type = 'png')
~~~
**Notes:**
- Shot number is auto-detected from current directory
- default image format is png
- **default xlabel is "Time [ms]' => time axis is rescaled to "ms"**
Pygolem files
=================
pygolem_lite.py
---------------------------
A basic module, it provides class Shot that can easily load data from the shot database [[link]](/SW/pygolem/pygolem_lite.py) . An example of use:
~~~ { .python }
from pygolem_lite import Shot
tvec, data = Shot(10322)['loop_voltage']
~~~
modules.py
------------------
A list of many useful functions that are used in the database [[link]](/SW/pygolem/modules.py) . For example: plotting functions, loading config files, code snippets for mako templates.
~~~ { .python }
def get_data(diagn, diagn_name, ylabel = "", xlabel = 'Time [ms]', xlim = [], ylim = [], \
tvec_rescale = 1e3, data_rescale = 1, integrated = False, columns = [], yscale = '', \
reduction = False, plot_limits = True, smoothing = 0, line_format = "", \
vlines = [], annotate = [], xgrid= False, ygrid = False, **kwargs ):
"""
Load data/diagn and return object that is used for plotting.
:var str/list/Data class diagn: Input can be string with name of diagn from main database, name of file with locally saved data on disk, list with [tvec,data]
:var str diagn_name: Name of diagnostics that will be used in legend, if diagn has more signal than name will be `name 0`, `name 1`, ...., or diagn_name can be list with name for each column
:var str ylabel: Title of ylabel
:var str xlabel: Title of xlabel
:var list xlim: Range for X axis, use [1.132 ,None] to define only one limit
:var list ylim: Range for Y axis, use [1.132 ,None] to define only one limit
:var double tvec_rescale: Rescale X axis to ms !! Be careful if the axis is not time in seconds !!
:var double data_rescale: Rescale Y axis
:var bool reduction: Try to reduce number of plotted points, automatically used for integrated signal
:var bool integrated: Integrate the input signal
:var list columns: Plot only the selected columns
:var str yscale: scale of y-axis, use `log`
:var bool plot_limits: If there is plasma, plot lines at start and end of the plasma
:var double smoothing: Smooth the signal
:var str line_format: Standard formating string
:var bool xgrid: Plot automatic xgrid in the plot
:var bool ygrid: Plot automatic ygrid in the plot
:var array xerr: error in X direction
:var array yerr: error in Y direction
:var **kwargs : keyword arguments to be passed to the standard plot() command, optional
this method accepts **kwargs as a standard :func:`matplotlib.pyplot.plot` function
Advanced:
:var list vlines : list of positions of vertical lines in plot (see historical analysis)
:var list annotate: list of dicts => plt.annotate(annot['text'], annot['pos'], rotation=annot['angle'] )
"""
def multiplot(data_all, title_name, file_name, figsize = (9,7), dpi = 100, orientation = 'vertical', file_type = 'png', reduction = False):
""" Plot several graphs to on multiplot. Use data `data_all` provided by function get_data !!
:var dict data_all: Input data or list of input data
:var str title_name: Title of the image
:var str file_name: Name of saved image (without filetype)
:var list figsize: Size of the output image
:var int dpi: DPI of output image
:var str orientation: Choose multiplot orientation: vertical/horizontal
:var str file_type: Image file type, recomended: png, svgz !! svgz is slow !!
:var bool reduction: Decrease number of plotted points => speed up plotting, use for all nice smooth lines !!
"""
~~~
config.py
-----------------
All tokamak constants that should be stable in time are listed in this module [[link]](/SW/pygolem/config.py)
MoinMoin.py
-----------------
A script that highlight code and generate a HTML from a given file [[link]](/SW/pygolem/MoinMoin.py)
code2html.py
------------------
A newer version of MoinMoin.py with better highlighting. Slightly slower.
utilities.py
----------------
Basic shared function library, load config, save config, cat, fix strings, detect None, ...
web.py
----------
All shared function for creation of web pages: create image, zoomed image, config, wiki link ,....
data
-----------------
CGI-like python script [[link]](/SW/pygolem/data). Transparently serve data from the main database to web interface.
An example of use:
[http://golem.fjfi.cvut.cz/cgi-bin/data/10322/loop_voltage](http://golem.fjfi.cvut.cz/cgi-bin/data/10322/loop_voltage)
golem_data.m
-----------------------
A script for matlab that uses web interface to download data [[link]](/SW/pygolem/golem_data.m). A simple example of use:
~~~ { .matlab }
[tvec, data, obj] = golem_data(10011, 'loop_voltage')
plot(tvec, data)
~~~
More examples are [here](/SW/DataProcessing#matlab---golem_data.m)
golem_data.py
-----------------------
A script for python that uses web interface to download data [[link]](/SW/pygolem/golem_data.py). A simple example of use:
~~~ { .python }
from golem_data import golem_data
from matplotlib.pyplot import *
obj = golem_data(10011, 'loop_voltage')
plot(obj1.tvec,obj1.data)
~~~
More examples are [here](/SW/DataProcessing#python---golem_data.py)
TODO
=====
[[link]](pygolem/TODO)
