LCLS Archiver restore¶
These examples show how single snapshots, and time series can be retreived from the archiver appliance.
Note that the times must always be in ISO 8601 format, UTC time (not local time).
In [1]:
Copied!
%pylab --no-import-all inline
%config InlineBackend.figure_format = 'retina'
%pylab --no-import-all inline
%config InlineBackend.figure_format = 'retina'
%pylab is deprecated, use %matplotlib inline and import the required libraries. Populating the interactive namespace from numpy and matplotlib
In [2]:
Copied!
from lcls_live.archiver import lcls_archiver_restore
from lcls_live import data_dir
import json
import os
# This is the main function
?lcls_archiver_restore
from lcls_live.archiver import lcls_archiver_restore
from lcls_live import data_dir
import json
import os
# This is the main function
?lcls_archiver_restore
Signature: lcls_archiver_restore( pvlist, isotime='2018-08-11T10:40:00.000-07:00', verbose=True, ) Docstring: Returns a dict of {'pvname':val} given a list of pvnames, at a time in ISO 8601 format, using the EPICS Archiver Appliance: https://slacmshankar.github.io/epicsarchiver_docs/userguide.html File: ~/Code/GitHub/lcls-live/lcls_live/archiver.py Type: function
Optional: off-site setup¶
In [3]:
Copied!
# Optional:
# Open an SSH tunnel in a terminal like:
# ssh -D 8080 <some user>@<some SLAC machine>
# And then set:
os.environ['http_proxy']='socks5h://localhost:8080'
os.environ['HTTPS_PROXY']='socks5h://localhost:8080'
os.environ['ALL_PROXY']='socks5h://localhost:8080'
# Optional:
# Open an SSH tunnel in a terminal like:
# ssh -D 8080 @
# And then set:
os.environ['http_proxy']='socks5h://localhost:8080'
os.environ['HTTPS_PROXY']='socks5h://localhost:8080'
os.environ['ALL_PROXY']='socks5h://localhost:8080'
Restore known PVs¶
In [4]:
Copied!
pvlist = [
'IRIS:LR20:130:MOTR_ANGLE',
'SOLN:IN20:121:BDES',
'QUAD:IN20:121:BDES',
'QUAD:IN20:122:BDES',
'ACCL:IN20:300:L0A_ADES',
'ACCL:IN20:300:L0A_PDES'
]
lcls_archiver_restore(pvlist, '2020-07-09T05:01:21.000000-07:00')
pvlist = [
'IRIS:LR20:130:MOTR_ANGLE',
'SOLN:IN20:121:BDES',
'QUAD:IN20:121:BDES',
'QUAD:IN20:122:BDES',
'ACCL:IN20:300:L0A_ADES',
'ACCL:IN20:300:L0A_PDES'
]
lcls_archiver_restore(pvlist, '2020-07-09T05:01:21.000000-07:00')
Requesting: http://lcls-archapp.slac.stanford.edu/retrieval/data/getDataAtTime?at=2020-07-09T05:01:21.000000-07:00&includeProxies=true
Out[4]:
{'IRIS:LR20:130:MOTR_ANGLE': 220.6020050048828,
'SOLN:IN20:121:BDES': 0.4745,
'QUAD:IN20:121:BDES': 0.0,
'QUAD:IN20:122:BDES': 0.0,
'ACCL:IN20:300:L0A_ADES': 58.5,
'ACCL:IN20:300:L0A_PDES': 0.0}
Get snapshot from a large list¶
Same as above, but for processing large amounts of data
In [5]:
Copied!
# Get list of PVs
fname = os.path.join(data_dir, 'classic/full_pvlist.json')
pvlist = json.load(open(fname))
pvlist[0:3]
# Get list of PVs
fname = os.path.join(data_dir, 'classic/full_pvlist.json')
pvlist = json.load(open(fname))
pvlist[0:3]
Out[5]:
['ACCL:IN20:350:FUDGE', 'ACCL:LI21:180:L1X_S_AV', 'ACCL:LI21:180:L1X_S_PV']
In [6]:
Copied!
# Simple filename naming
def snapshot_filename(isotime):
return 'epics_snapshot_'+isotime+'.json'
# Simple filename naming
def snapshot_filename(isotime):
return 'epics_snapshot_'+isotime+'.json'
In [7]:
Copied!
times = ['2022-03-06T15:21:15.000000-08:00']
lcls_archiver_restore(pvlist[0:10], times[0])
times = ['2022-03-06T15:21:15.000000-08:00']
lcls_archiver_restore(pvlist[0:10], times[0])
Requesting: http://lcls-archapp.slac.stanford.edu/retrieval/data/getDataAtTime?at=2022-03-06T15:21:15.000000-08:00&includeProxies=true
Out[7]:
{'ACCL:IN20:350:FUDGE': 1.0085738911621906,
'ACCL:LI21:180:L1X_S_AV': 21.00412087581118,
'ACCL:LI21:180:L1X_S_PV': -159.84138473850933,
'ACCL:LI21:1:FUDGE': 1.0100564939191123,
'ACCL:LI21:1:L1S_S_PV': -19.680153655018337,
'ACCL:LI22:1:FUDGE': 0.9296999422680153,
'ACCL:LI24:100:KLY_PDES': 15.11477461877093,
'ACCL:LI24:200:KLY_PDES': -28.841842462210366,
'ACCL:LI24:300:KLY_PDES': -72.63314256041122,
'ACCL:LI25:1:FUDGE': 0.9872969769907864}
In [8]:
Copied!
%%time
# Make multiple files
root = './data/'
for t in times:
newdata = lcls_archiver_restore(pvlist, t)
fname = os.path.join(root, snapshot_filename(t))
with open(fname, 'w') as f:
f.write(json.dumps(newdata))
print('Written:', fname)
%%time
# Make multiple files
root = './data/'
for t in times:
newdata = lcls_archiver_restore(pvlist, t)
fname = os.path.join(root, snapshot_filename(t))
with open(fname, 'w') as f:
f.write(json.dumps(newdata))
print('Written:', fname)
Requesting: http://lcls-archapp.slac.stanford.edu/retrieval/data/getDataAtTime?at=2022-03-06T15:21:15.000000-08:00&includeProxies=true Warning: Missing PV: BTRM:LTU1:280:BDES Warning: Missing PV: SHTR:LR20:200:UV_STS Warning: Missing PV: XCOR:LTU1:288:BDES Warning: Missing PV: XCOR:LTU1:448:BDES Written: ./data/epics_snapshot_2022-03-06T15:21:15.000000-08:00.json CPU times: user 20.8 ms, sys: 5.85 ms, total: 26.7 ms Wall time: 2min
Get history of a single PV¶
This package also has a couple functions for getting the time history data of a pv.
The first, lcls_archiver_history, returns the raw data
In [9]:
Copied!
from lcls_live.archiver import lcls_archiver_history, lcls_archiver_history_dataframe
t_start = '2020-07-09T05:01:15.000000-07:00'
t_end = '2020-07-09T05:03:00.000000-07:00'
secs, vals = lcls_archiver_history('SOLN:IN20:121:BDES', start=t_start, end=t_end)
secs[0:5], vals[0:5]
from lcls_live.archiver import lcls_archiver_history, lcls_archiver_history_dataframe
t_start = '2020-07-09T05:01:15.000000-07:00'
t_end = '2020-07-09T05:03:00.000000-07:00'
secs, vals = lcls_archiver_history('SOLN:IN20:121:BDES', start=t_start, end=t_end)
secs[0:5], vals[0:5]
http://lcls-archapp.slac.stanford.edu/retrieval/data/getData.json?pv=SOLN:IN20:121:BDES&from=2020-07-09T05:01:15.000000-07:00&to=2020-07-09T05:03:00.000000-07:00
Out[9]:
([1594296071, 1594296075, 1594296079, 1594296084, 1594296088], [0.47, 0.47224999999999995, 0.4745, 0.47675, 0.479])
More convenient, is to format this as a pandas dataframe.
In [10]:
Copied!
?lcls_archiver_history_dataframe
?lcls_archiver_history_dataframe
Signature: lcls_archiver_history_dataframe(pvname, **kwargs) Docstring: Same as lcls_archiver_history, but returns a dataframe with the index as the time. File: ~/Code/GitHub/lcls-live/lcls_live/archiver.py Type: function
In [11]:
Copied!
df1 = lcls_archiver_history_dataframe('YAGS:IN20:241:YRMS', start=t_start, end=t_end)
df1[0:5]
df1 = lcls_archiver_history_dataframe('YAGS:IN20:241:YRMS', start=t_start, end=t_end)
df1[0:5]
http://lcls-archapp.slac.stanford.edu/retrieval/data/getData.json?pv=YAGS:IN20:241:YRMS&from=2020-07-09T05:01:15.000000-07:00&to=2020-07-09T05:03:00.000000-07:00
Out[11]:
| YAGS:IN20:241:YRMS | |
|---|---|
| time | |
| 2020-07-09 11:59:48 | 433.009985 |
| 2020-07-09 12:01:15 | 292.644774 |
| 2020-07-09 12:01:19 | 262.210378 |
| 2020-07-09 12:01:23 | 226.717519 |
| 2020-07-09 12:01:28 | 230.642901 |
In [12]:
Copied!
# Pandas has convenient plotting
df1.plot()
# Pandas has convenient plotting
df1.plot()
Out[12]:
<AxesSubplot:xlabel='time'>
Aligning the history of two PVs¶
The returned data will not necessarily be time-aligned. Here we will use Pandas' interpolate capabilities to fill in missing data.
In [13]:
Copied!
import pandas as pd
import pandas as pd
In [14]:
Copied!
# Try another PV. This one was smoothy scanned
df2 = lcls_archiver_history_dataframe('SOLN:IN20:121:BDES', start=t_start, end=t_end)
df2
# Try another PV. This one was smoothy scanned
df2 = lcls_archiver_history_dataframe('SOLN:IN20:121:BDES', start=t_start, end=t_end)
df2
http://lcls-archapp.slac.stanford.edu/retrieval/data/getData.json?pv=SOLN:IN20:121:BDES&from=2020-07-09T05:01:15.000000-07:00&to=2020-07-09T05:03:00.000000-07:00
Out[14]:
| SOLN:IN20:121:BDES | |
|---|---|
| time | |
| 2020-07-09 12:01:11 | 0.470000 |
| 2020-07-09 12:01:15 | 0.472250 |
| 2020-07-09 12:01:19 | 0.474500 |
| 2020-07-09 12:01:24 | 0.476750 |
| 2020-07-09 12:01:28 | 0.479000 |
| 2020-07-09 12:01:32 | 0.481250 |
| 2020-07-09 12:01:37 | 0.483500 |
| 2020-07-09 12:01:41 | 0.485750 |
| 2020-07-09 12:01:46 | 0.488000 |
| 2020-07-09 12:01:50 | 0.465016 |
In [15]:
Copied!
df2.plot()
df2.plot()
Out[15]:
<AxesSubplot:xlabel='time'>
In [16]:
Copied!
# Notice that some data are taken at the same time, others are not
df4 = pd.concat([df1, df2], axis=1)
df4
# Notice that some data are taken at the same time, others are not
df4 = pd.concat([df1, df2], axis=1)
df4
Out[16]:
| YAGS:IN20:241:YRMS | SOLN:IN20:121:BDES | |
|---|---|---|
| time | ||
| 2020-07-09 11:59:48 | 433.009985 | NaN |
| 2020-07-09 12:01:11 | NaN | 0.470000 |
| 2020-07-09 12:01:15 | 292.644774 | 0.472250 |
| 2020-07-09 12:01:19 | 262.210378 | 0.474500 |
| 2020-07-09 12:01:23 | 226.717519 | NaN |
| 2020-07-09 12:01:24 | NaN | 0.476750 |
| 2020-07-09 12:01:28 | 230.642901 | 0.479000 |
| 2020-07-09 12:01:32 | 243.905289 | 0.481250 |
| 2020-07-09 12:01:37 | NaN | 0.483500 |
| 2020-07-09 12:01:41 | 342.439957 | 0.485750 |
| 2020-07-09 12:01:45 | 427.501542 | NaN |
| 2020-07-09 12:01:46 | NaN | 0.488000 |
| 2020-07-09 12:01:49 | 486.459644 | NaN |
| 2020-07-09 12:01:50 | NaN | 0.465016 |
In [17]:
Copied!
# This will fill in the missing values, and drop trailing NaNs
df5 = df4.interpolate().dropna()
df5
# This will fill in the missing values, and drop trailing NaNs
df5 = df4.interpolate().dropna()
df5
Out[17]:
| YAGS:IN20:241:YRMS | SOLN:IN20:121:BDES | |
|---|---|---|
| time | ||
| 2020-07-09 12:01:11 | 362.827380 | 0.470000 |
| 2020-07-09 12:01:15 | 292.644774 | 0.472250 |
| 2020-07-09 12:01:19 | 262.210378 | 0.474500 |
| 2020-07-09 12:01:23 | 226.717519 | 0.475625 |
| 2020-07-09 12:01:24 | 228.680210 | 0.476750 |
| 2020-07-09 12:01:28 | 230.642901 | 0.479000 |
| 2020-07-09 12:01:32 | 243.905289 | 0.481250 |
| 2020-07-09 12:01:37 | 293.172623 | 0.483500 |
| 2020-07-09 12:01:41 | 342.439957 | 0.485750 |
| 2020-07-09 12:01:45 | 427.501542 | 0.486875 |
| 2020-07-09 12:01:46 | 456.980593 | 0.488000 |
| 2020-07-09 12:01:49 | 486.459644 | 0.476508 |
| 2020-07-09 12:01:50 | 486.459644 | 0.465016 |
In [18]:
Copied!
# make a plot
DF = df5[:-2] # The last two points are outside the main scan.
k1 = 'SOLN:IN20:121:BDES'
k2 = 'YAGS:IN20:241:YRMS'
plt.xlabel(k1)
plt.ylabel(k2)
plt.title(t_start+'\n to '+t_end)
plt.scatter(DF[k1], DF[k2], marker='.', color='black')
# make a plot
DF = df5[:-2] # The last two points are outside the main scan.
k1 = 'SOLN:IN20:121:BDES'
k2 = 'YAGS:IN20:241:YRMS'
plt.xlabel(k1)
plt.ylabel(k2)
plt.title(t_start+'\n to '+t_end)
plt.scatter(DF[k1], DF[k2], marker='.', color='black')
Out[18]:
<matplotlib.collections.PathCollection at 0x14e3e4340>
This easily extends to a list¶
In [19]:
Copied!
pvlist = ['SOLN:IN20:121:BDES', 'YAGS:IN20:241:XRMS', 'YAGS:IN20:241:YRMS']
dflist = []
for pvname in pvlist:
dflist.append(lcls_archiver_history_dataframe(pvname, start=t_start, end=t_end))
df6 = pd.concat(dflist, axis=1).interpolate().dropna()
pvlist = ['SOLN:IN20:121:BDES', 'YAGS:IN20:241:XRMS', 'YAGS:IN20:241:YRMS']
dflist = []
for pvname in pvlist:
dflist.append(lcls_archiver_history_dataframe(pvname, start=t_start, end=t_end))
df6 = pd.concat(dflist, axis=1).interpolate().dropna()
http://lcls-archapp.slac.stanford.edu/retrieval/data/getData.json?pv=SOLN:IN20:121:BDES&from=2020-07-09T05:01:15.000000-07:00&to=2020-07-09T05:03:00.000000-07:00 http://lcls-archapp.slac.stanford.edu/retrieval/data/getData.json?pv=YAGS:IN20:241:XRMS&from=2020-07-09T05:01:15.000000-07:00&to=2020-07-09T05:03:00.000000-07:00 http://lcls-archapp.slac.stanford.edu/retrieval/data/getData.json?pv=YAGS:IN20:241:YRMS&from=2020-07-09T05:01:15.000000-07:00&to=2020-07-09T05:03:00.000000-07:00
In [20]:
Copied!
df6
df6
Out[20]:
| SOLN:IN20:121:BDES | YAGS:IN20:241:XRMS | YAGS:IN20:241:YRMS | |
|---|---|---|---|
| time | |||
| 2020-07-09 12:01:11 | 0.470000 | 393.366373 | 362.827380 |
| 2020-07-09 12:01:15 | 0.472250 | 325.602016 | 292.644774 |
| 2020-07-09 12:01:19 | 0.474500 | 297.144936 | 262.210378 |
| 2020-07-09 12:01:23 | 0.475625 | 269.941236 | 226.717519 |
| 2020-07-09 12:01:24 | 0.476750 | 262.813117 | 228.680210 |
| 2020-07-09 12:01:28 | 0.479000 | 255.684997 | 230.642901 |
| 2020-07-09 12:01:32 | 0.481250 | 246.747285 | 243.905289 |
| 2020-07-09 12:01:36 | 0.482375 | 266.423525 | 276.750178 |
| 2020-07-09 12:01:37 | 0.483500 | 279.129956 | 309.595068 |
| 2020-07-09 12:01:41 | 0.485750 | 291.836386 | 342.439957 |
| 2020-07-09 12:01:45 | 0.486875 | 335.522076 | 427.501542 |
| 2020-07-09 12:01:46 | 0.488000 | 367.396032 | 456.980593 |
| 2020-07-09 12:01:49 | 0.476508 | 399.269987 | 486.459644 |
| 2020-07-09 12:01:50 | 0.465016 | 399.269987 | 486.459644 |
In [21]:
Copied!
DF = df6[:-2] # Drop the last two that are unrelated to the scan
k1 = 'SOLN:IN20:121:BDES'
k2 = 'YAGS:IN20:241:XRMS'
k3 = 'YAGS:IN20:241:YRMS'
plt.xlabel(k1+' (kG-m)')
plt.ylabel('Measurement (um)')
plt.title(t_start+'\n to '+t_end)
X1 = DF[k1]
X2 = DF[k2]
X3 = DF[k3]
plt.scatter(X1, X2, marker='x', color='blue', label=k2)
plt.scatter(X1, X3, marker='x', color='green', label=k3)
plt.legend()
DF = df6[:-2] # Drop the last two that are unrelated to the scan
k1 = 'SOLN:IN20:121:BDES'
k2 = 'YAGS:IN20:241:XRMS'
k3 = 'YAGS:IN20:241:YRMS'
plt.xlabel(k1+' (kG-m)')
plt.ylabel('Measurement (um)')
plt.title(t_start+'\n to '+t_end)
X1 = DF[k1]
X2 = DF[k2]
X3 = DF[k3]
plt.scatter(X1, X2, marker='x', color='blue', label=k2)
plt.scatter(X1, X3, marker='x', color='green', label=k3)
plt.legend()
Out[21]:
<matplotlib.legend.Legend at 0x14e453d30>
