This page was generated from docs/source/tutorial/quick-start.ipynb. Interactive online version: Binder badge Slideshow: Binder badge

Quickstart of SuPy

This quickstart demonstrates the essential and simplest workflow of supy in SUEWS simulation:

  1. load input files
  2. run simulation
  3. examine results

More advanced use of supy are available in the tutorials

Before start, we need to load the following necessary packages.

[1]:
import matplotlib.pyplot as plt
import supy as sp
import pandas as pd
import numpy as np
from pathlib import Path
get_ipython().run_line_magic('matplotlib', 'inline')

# produce high-quality figures, which can also be set as one of ['svg', 'pdf', 'retina', 'png']
# 'svg' produces high quality vector figures
%config InlineBackend.figure_format = 'svg'
[2]:
sp.show_version()
supy: 2019.8.30dev
supy_driver: 2019a4

Load input files

For existing SUEWS users:

First, a path to SUEWS RunControl.nml should be specified, which will direct supy to locate input files.

[3]:
path_runcontrol = Path('../sample_run') / 'RunControl.nml'
[4]:
df_state_init = sp.init_supy(path_runcontrol)
INFO:root:All cache cleared.

A sample df_state_init looks below (note that .T is used here to a nicer tableform view):

[5]:
df_state_init.filter(like='method').T
[5]:
grid 98
var ind_dim
aerodynamicresistancemethod 0 2
evapmethod 0 2
emissionsmethod 0 2
netradiationmethod 0 3
roughlenheatmethod 0 2
roughlenmommethod 0 2
smdmethod 0 0
stabilitymethod 0 3
storageheatmethod 0 1
waterusemethod 0 0

Following the convention of SUEWS, supy loads meteorological forcing (met-forcing) files at the grid level.

[6]:
grid = df_state_init.index[0]
df_forcing = sp.load_forcing_grid(path_runcontrol, grid)
INFO:root:All cache cleared.

For new users to SUEWS/SuPy:

To ease the input file preparation, a helper function load_SampleData is provided to get the sample input for SuPy simulations

[7]:
df_state_init, df_forcing = sp.load_SampleData()
INFO:root:All cache cleared.

Overview of SuPy input

df_state_init

df_state_init includes model Initial state consisting of:

  • surface characteristics (e.g., albedo, emissivity, land cover fractions, etc.; full details refer to SUEWS documentation)
  • model configurations (e.g., stability; full details refer to SUEWS documentation)

Detailed description of variables in df_state_init refers to SuPy input

Surface land cover fraction information in the sample input dataset:

[8]:
df_state_init.loc[:,['bldgh','evetreeh','dectreeh']]
[8]:
var bldgh dectreeh evetreeh
ind_dim 0 0 0
grid
98 22.0 13.1 13.1
[9]:
df_state_init.filter(like='sfr')
[9]:
var sfr
ind_dim (0,) (1,) (2,) (3,) (4,) (5,) (6,)
grid
98 0.43 0.38 0.001 0.019 0.029 0.001 0.14

df_forcing

df_forcing includes meteorological and other external forcing information.

Detailed description of variables in df_forcing refers to SuPy input.

Below is an overview of forcing variables of the sample data set used in the following simulations.

[10]:
list_var_forcing = [
    'kdown',
    'Tair',
    'RH',
    'pres',
    'U',
    'rain',
]
dict_var_label = {
    'kdown': 'Incoming Solar\n Radiation ($ \mathrm{W \ m^{-2}}$)',
    'Tair': 'Air Temperature ($^{\circ}}$C)',
    'RH': r'Relative Humidity (%)',
    'pres': 'Air Pressure (hPa)',
    'rain': 'Rainfall (mm)',
    'U': 'Wind Speed (m $\mathrm{s^{-1}}$)'
}
df_plot_forcing_x = df_forcing.loc[:, list_var_forcing].copy().shift(
    -1).dropna(how='any')
df_plot_forcing = df_plot_forcing_x.resample('1h').mean()
df_plot_forcing['rain'] = df_plot_forcing_x['rain'].resample('1h').sum()

axes = df_plot_forcing.plot(
    subplots=True,
    figsize=(8, 12),
    legend=False,
)
fig = axes[0].figure
fig.tight_layout()
fig.autofmt_xdate(bottom=0.2, rotation=0, ha='center')
for ax, var in zip(axes, list_var_forcing):
    ax.set_ylabel(dict_var_label[var])
../_images/tutorial_quick-start_21_0.svg

Modification of SuPy input

Given pandas.DataFrame as the core data structure of SuPy, all operations, including modification, output, demonstration, etc., on SuPy inputs (df_state_init and df_forcing) can be done using pandas-based functions/methods.

Specifically, for modification, the following operations are essential:

locating data

Data can be located in two ways, namely: 1. by name via `.loc <http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#selection-by-label>`__; 2. by position via `.iloc <http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#selection-by-position>`__.

[11]:
# view the surface fraction variable: `sfr`
df_state_init.loc[:,'sfr']
[11]:
ind_dim (0,) (1,) (2,) (3,) (4,) (5,) (6,)
grid
98 0.43 0.38 0.001 0.019 0.029 0.001 0.14
[12]:
# view the second row of `df_forcing`, which is a pandas Series
df_forcing.iloc[1]
[12]:
iy       2012.000000
id          1.000000
it          0.000000
imin       10.000000
qn       -999.000000
qh       -999.000000
qe       -999.000000
qs       -999.000000
qf       -999.000000
U           4.515000
RH         85.463333
Tair       11.773750
pres     1001.512500
rain        0.000000
kdown       0.153333
snow     -999.000000
ldown    -999.000000
fcld     -999.000000
Wuh      -999.000000
xsmd     -999.000000
lai      -999.000000
kdiff    -999.000000
kdir     -999.000000
wdir     -999.000000
isec        0.000000
Name: 2012-01-01 00:10:00, dtype: float64
[13]:
# view a particular position of `df_forcing`, which is a value
df_forcing.iloc[8,9]
[13]:
4.455

setting new values

Setting new values is very straightforward: after locating the variables/data to modify, just set the new values accordingly:

[14]:
# modify surface fractions
df_state_init.loc[:,'sfr']=[.1,.1,.2,.3,.25,.05,0]
# check the updated values
df_state_init.loc[:,'sfr']
[14]:
ind_dim (0,) (1,) (2,) (3,) (4,) (5,) (6,)
grid
98 0.1 0.1 0.2 0.3 0.25 0.05 0.0

Run simulations

Once met-forcing (via df_forcing) and initial conditions (via df_state_init) are loaded in, we call sp.run_supy to conduct a SUEWS simulation, which will return two pandas DataFrames: df_output and df_state.

[15]:
df_output, df_state_final = sp.run_supy(df_forcing, df_state_init)
INFO:root:====================
INFO:root:Simulation period:
INFO:root:  Start: 2012-01-01 00:05:00
INFO:root:  End: 2013-01-01 00:00:00
INFO:root:
INFO:root:No. of grids: 1
INFO:root:SuPy is running in serial mode
INFO:root:Execution time: 3.2 s
INFO:root:====================

df_output

df_output is an ensemble output collection of major SUEWS output groups, including:

  • SUEWS: the essential SUEWS output variables
  • DailyState: variables of daily state information
  • snow: snow output variables (effective when snowuse = 1 set in df_state_init)

Detailed description of variables in df_output refers to SuPy output

[16]:
df_output.columns.levels[0]
[16]:
Index(['SUEWS', 'snow', 'RSL', 'DailyState'], dtype='object', name='group')

df_state_final

df_state_final is a DataFrame for holding:

  1. all model states if save_state is set to True when calling sp.run_supy and supy may run significantly slower for a large simulation;
  2. or, only the final state if save_state is set to False (the default setting) in which mode supy has a similar performance as the standalone compiled SUEWS executable.

Entries in df_state_final have the same data structure as df_state_init and can thus be used for other SUEWS simulations staring at the timestamp as in df_state_final.

Detailed description of variables in df_state_final refers to SuPy output

[17]:
df_state_final.T.head()
[17]:
datetime 2012-01-01 00:05:00 2013-01-01 00:05:00
grid 98 98
var ind_dim
ah_min (0,) 15.0 15.0
(1,) 15.0 15.0
ah_slope_cooling (0,) 2.7 2.7
(1,) 2.7 2.7
ah_slope_heating (0,) 2.7 2.7

Examine results

Thanks to the functionality inherited from pandas and other packages under the PyData stack, compared with the standard SUEWS simulation workflow, supy enables more convenient examination of SUEWS results by statistics calculation, resampling, plotting (and many more).

Ouptut structure

df_output is organised with MultiIndex (grid,timestamp) and (group,varaible) as index and columns, respectively.

[18]:
df_output.head()
[18]:
group SUEWS ... DailyState
var Kdown Kup Ldown Lup Tsurf QN QF QS QH QE ... DensSnow_Paved DensSnow_Bldgs DensSnow_EveTr DensSnow_DecTr DensSnow_Grass DensSnow_BSoil DensSnow_Water a1 a2 a3
grid datetime
98 2012-01-01 00:05:00 0.153333 0.021237 344.310184 372.417244 11.775859 -27.974963 40.569300 -45.253674 57.807360 0.040651 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2012-01-01 00:10:00 0.153333 0.021237 344.310184 372.417244 11.775859 -27.974963 39.719681 -45.070905 56.775225 0.040398 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2012-01-01 00:15:00 0.153333 0.021237 344.310184 372.417244 11.775859 -27.974963 38.870062 -44.895750 55.750704 0.040145 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2012-01-01 00:20:00 0.153333 0.021237 344.310184 372.417244 11.775859 -27.974963 38.020443 -44.727894 54.733480 0.039895 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2012-01-01 00:25:00 0.153333 0.021237 344.310184 372.417244 11.775859 -27.974963 37.170824 -44.567032 53.723248 0.039645 ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN

5 rows × 340 columns

Here we demonstrate several typical scenarios for SUEWS results examination.

The essential SUEWS output collection is extracted as a separate variable for easier processing in the following sections. More advanced slicing techniques are available in pandas documentation.

[19]:
df_output_suews = df_output['SUEWS']

Statistics Calculation

We can use .describe() method for a quick overview of the key surface energy balance budgets.

[20]:
df_output_suews.loc[:, ['QN', 'QS', 'QH', 'QE', 'QF']].describe()
[20]:
var QN QS QH QE QF
count 105408.000000 105408.000000 105408.000000 105408.000000 105408.000000
mean 39.319914 -15.810252 88.755915 45.857651 79.068259
std 130.797388 53.953592 69.057335 54.363737 30.855099
min -86.212629 -87.482114 -114.375930 0.000081 26.506045
25% -42.028676 -48.084784 41.334831 1.266435 50.520548
50% -25.694495 -40.948527 75.221473 22.980817 82.815455
75% 73.254869 -2.433109 126.971057 75.607932 104.577731
max 662.453669 239.033524 371.051513 378.152626 162.947824

Plotting

Basic example

Plotting is very straightforward via the .plot method bounded with pandas.DataFrame. Note the usage of loc for to slices of the output DataFrame.

[21]:
# a dict for better display variable names
dict_var_disp = {
    'QN': '$Q^*$',
    'QS': r'$\Delta Q_S$',
    'QE': '$Q_E$',
    'QH': '$Q_H$',
    'QF': '$Q_F$',
    'Kdown': r'$K_{\downarrow}$',
    'Kup': r'$K_{\uparrow}$',
    'Ldown': r'$L_{\downarrow}$',
    'Lup': r'$L_{\uparrow}$',
    'Rain': '$P$',
    'Irr': '$I$',
    'Evap': '$E$',
    'RO': '$R$',
    'TotCh': '$\Delta S$',
}

Quick look at the simulation results:

[22]:
ax_output = df_output_suews\
    .loc[grid]\
    .loc['2012 6 1':'2012 6 7',
         ['QN', 'QS', 'QE', 'QH', 'QF']]\
    .rename(columns=dict_var_disp)\
    .plot()
ax_output.set_xlabel('Date')
ax_output.set_ylabel('Flux ($ \mathrm{W \ m^{-2}}$)')
ax_output.legend()
[22]:
<matplotlib.legend.Legend at 0x7f8ec14566a0>
../_images/tutorial_quick-start_51_1.svg

More examples

Below is a more complete example for examination of urban energy balance over the whole summer (June to August).

[23]:
# energy balance
ax_output = df_output_suews.loc[grid]\
    .loc['2012 6':'2012 8', ['QN', 'QS', 'QE', 'QH', 'QF']]\
    .rename(columns=dict_var_disp)\
    .plot(
        figsize=(10, 3),
        title='Surface Energy Balance',
    )
ax_output.set_xlabel('Date')
ax_output.set_ylabel('Flux ($ \mathrm{W \ m^{-2}}$)')
ax_output.legend()
[23]:
<matplotlib.legend.Legend at 0x7f8ed08c3278>
../_images/tutorial_quick-start_53_1.svg

Resampling

The suggested runtime/simulation frequency of SUEWS is 300 s, which usually results a large output and may be over-weighted for storage and analysis. Also, you may feel apparent slowdown in producing the above figure as a large amount of data were used for the plotting. To slim down the result size for analysis and output, we can resample the default output very easily.

[24]:
rsmp_1d = df_output_suews.loc[grid].resample('1d')
# daily mean values
df_1d_mean = rsmp_1d.mean()
# daily sum values
df_1d_sum = rsmp_1d.sum()

We can then re-examine the above energy balance at hourly scale and plotting will be significantly faster.

[25]:
# energy balance
ax_output = df_1d_mean\
    .loc[:, ['QN', 'QS', 'QE', 'QH', 'QF']]\
    .rename(columns=dict_var_disp)\
    .plot(
            figsize=(10, 3),
            title='Surface Energy Balance',
        )
ax_output.set_xlabel('Date')
ax_output.set_ylabel('Flux ($ \mathrm{W \ m^{-2}}$)')
ax_output.legend()
[25]:
<matplotlib.legend.Legend at 0x7f8ec1623908>
../_images/tutorial_quick-start_57_1.svg

Then we use the hourly results for other analyses.

[26]:
# radiation balance
ax_output = df_1d_mean\
    .loc[:, ['QN', 'Kdown', 'Kup', 'Ldown', 'Lup']]\
    .rename(columns=dict_var_disp)\
    .plot(
        figsize=(10, 3),
        title='Radiation Balance',
    )
ax_output.set_xlabel('Date')
ax_output.set_ylabel('Flux ($ \mathrm{W \ m^{-2}}$)')
ax_output.legend()
[26]:
<matplotlib.legend.Legend at 0x7f8eb149a0b8>
../_images/tutorial_quick-start_59_1.svg
[27]:
# water balance
ax_output = df_1d_sum\
    .loc[:, ['Rain', 'Irr', 'Evap', 'RO', 'TotCh']]\
    .rename(columns=dict_var_disp)\
    .plot(
        figsize=(10, 3),
        title='Surface Water Balance',
    )
ax_output.set_xlabel('Date')
ax_output.set_ylabel('Water amount (mm)')
ax_output.legend()
[27]:
<matplotlib.legend.Legend at 0x7f8ef208bc18>
../_images/tutorial_quick-start_60_1.svg

Get an overview of partitioning in energy and water balance at monthly scales:

[28]:
# get a monthly Resampler
df_plot=df_output_suews.loc[grid].copy()
df_plot.index=df_plot.index.set_names('Month')
rsmp_1M = df_plot\
    .shift(-1)\
    .dropna(how='all')\
    .resample('1M', kind='period')
# mean values
df_1M_mean = rsmp_1M.mean()
# sum values
df_1M_sum = rsmp_1M.sum()

[29]:
# month names
name_mon = [x.strftime('%b') for x in rsmp_1M.groups]
# create subplots showing two panels together
fig, axes = plt.subplots(2, 1, sharex=True)
# surface energy balance
df_1M_mean\
    .loc[:, ['QN', 'QS', 'QE', 'QH', 'QF']]\
    .rename(columns=dict_var_disp)\
    .plot(
        ax=axes[0],  # specify the axis for plotting
        figsize=(10, 6),  # specify figure size
        title='Surface Energy Balance',
        kind='bar',
    )
# surface water balance
df_1M_sum\
    .loc[:, ['Rain', 'Irr', 'Evap', 'RO', 'TotCh']]\
    .rename(columns=dict_var_disp)\
    .plot(
        ax=axes[1],  # specify the axis for plotting
        title='Surface Water Balance',
        kind='bar'
    )

# annotations
axes[0].set_ylabel('Mean Flux ($ \mathrm{W \ m^{-2}}$)')
axes[0].legend()
axes[1].set_xlabel('Month')
axes[1].set_ylabel('Total Water Amount (mm)')
axes[1].xaxis.set_ticklabels(name_mon, rotation=0)
axes[1].legend()
[29]:
<matplotlib.legend.Legend at 0x7f8ec195a9b0>
../_images/tutorial_quick-start_63_1.svg

Output

The supy output can be saved as txt files for further analysis using supy function save_supy.

[30]:
list_path_save = sp.save_supy(df_output,df_state_final, path_runcontrol=path_runcontrol)
[31]:
for file_out in list_path_save:
    print(file_out.name)
Kc98_2012_SUEWS_5.txt
Kc98_2012_snow_5.txt
Kc98_2012_RSL_5.txt
Kc98_2012_DailyState.txt
Kc98_2012_SUEWS_60.txt
Kc98_2012_snow_60.txt
Kc98_2012_RSL_60.txt
InitialConditionsKc98_2013_EndofRun.nml