Skip to content
Snippets Groups Projects
Unverified Commit a4cb0a38 authored by David Hoese's avatar David Hoese
Browse files

Add display names to archive info

parent bfeb75fe
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
metobsapi/templates/files_index.html
+ 54
− 243
View file @ a4cb0a38
......@@ -151,8 +151,11 @@
</td>
<td style='padding: 6px;'>
(Required) Colon separated list of product attributes to query for.
streams are specified as a colon separated list of '&#60site&#62.&#60instrument&#62.&#60stem_name&#62.&#60product_level&#62[.&#60version_number&#62].' If the version number isn't specified, the most recent one is used. <b>Note</b>: level and version is abbreviated
(i.e. level_00 becomes l00 and version_00 becmes v00).
streams are specified as a colon separated list of '&#60site&#62.&#60instrument&#62.&#60product_id&#62.&#60product_level&#62.&#60version_number&#62'.
The version number may be specificed as '*' to choose the most recent version.
The product ID may also be specified as '*' to choose all products for a given level.
<br><b>Note</b>: level and version is abbreviated
(i.e. level_00 becomes l00 and version_00 becomes v00).
Valid streams quantifiers for each site/instrument are documented below.
</td>
</tr>
......@@ -177,263 +180,71 @@
AOSS Tower:
</h3>
<table class="table table-striped table-bordered">
<tr>
<td class="col-md-3" style='padding: 6px;text-align: center;'>
<b>Stream Name</b>
</td>
<td class="col-md-3" style='padding: 6px; text-align: center'>
<b>File Pattern</b>
</td>
<td class="col-md-6" style='padding: 6px; text-align: center'>
<b>Description</b>
</td>
</tr>
{% for stream_id in instrument_streams['aoss.tower'] | sort %}
<tr>
<td class="col-md-6" style='padding: 6px;text-align: center;'>
<b>Stem Name:</b>
</td>
<td class="col-md-3" style='padding: 6px; text-align: center'>
<b>Product Level:</b>
</td>
<td class="col-md-3" style='padding: 6px; text-align: center'>
<b>File Pattern</b>
</td>
</tr>
<tr>
<td style='padding: 6px;'>
ascii
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
rig_tower.YYYY-MM-DD.ascii
</td>
<td style='padding: 6px;'>{{ stream_id }}</td>
{% if 'file_pattern' in archive_info[stream_id] %}
<td style='padding: 6px;'>{{ archive_info[stream_id]['file_pattern'] }}</td>
{% else %}
<td style='padding: 6px;'>N/A</td>
{% endif %}
{% if 'description' in archive_info[stream_id] %}
<td style='padding: 6px;'>{{ archive_info[stream_id]['description'] }}</td>
{% else %}
<td style='padding: 6px;'></td>
{% endif %}
</tr>
{% endfor %}
</table>
<h3>
Versions:
</h3>
<p>
Currently, there is only one version 'version_00' and it has been used from 05/28/2003-present
</p>
<h3>
AOSS AERI:
</h3>
<table class="table table-striped table-bordered">
<tr>
<td class="col-md-3" style='padding: 6px;text-align: center;'>
<b>Stream Name</b>
</td>
<td class="col-md-3" style='padding: 6px; text-align: center'>
<b>File Pattern</b>
</td>
<td class="col-md-6" style='padding: 6px; text-align: center'>
<b>Description</b>
</td>
</tr>
{% for stream_id in instrument_streams['aoss.aeri'] | sort %}
<tr>
<td class="col-md-6" style='padding: 6px;text-align: center;'>
<b>Stem Name:</b>
</td>
<td class="col-md-3" style='padding: 6px; text-align: center'>
<b>Product Level:</b>
</td>
<td class="col-md-3" style='padding: 6px; text-align: center'>
<b>File Pattern</b>
</td>
</tr>
<tr>
<td style='padding: 6px;'>
PAR
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDD.PAR
</td>
</tr>
<tr>
<td style='padding: 6px;'>
QC
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDD.QC
</td>
</tr>
<tr>
<td style='padding: 6px;'>
SUM
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDD.SUM
</td>
</tr>
<tr>
<td style='padding: 6px;'>
B1_CXS
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDB1.CXS
</td>
</tr>
<tr>
<td style='padding: 6px;'>
B1_UVS
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDB1.UVS
</td>
</tr>
<tr>
<td style='padding: 6px;'>
B2_CXS
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDB2.CXS
</td>
</tr>
<tr>
<td style='padding: 6px;'>
B2_UVS
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDB2.UVS
</td>
</tr>
<tr>
<td style='padding: 6px;'>
C1_RNC
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDC1.RNC
</td>
</tr>
<tr>
<td style='padding: 6px;'>
C2_RNC
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDC2.RNC
</td>
</tr>
<tr>
<td style='padding: 6px;'>
F1_CSV
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDF1.CSV
</td>
</tr>
<tr>
<td style='padding: 6px;'>
F1_CXS
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDF1.CXS
</td>
</tr>
<tr>
<td style='padding: 6px;'>
F1_UVS
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDF1.UVS
</td>
</tr>
<tr>
<td style='padding: 6px;'>
F2_CSV
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDF2.CSV
</td>
</tr>
<tr>
<td style='padding: 6px;'>
F2_CXS
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDF2.CXS
</td>
</tr>
<tr>
<td style='padding: 6px;'>
F2_UVS
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
YYMMDDF2.UVS
</td>
</tr>
<tr>
<td style='padding: 6px;'>
R_SCR
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
AESITTER.SCR
</td>
</tr>
<tr>
<td style='padding: 6px;'>
E_SCR
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
RADIANCE.SCR
</td>
</tr>
<tr>
<td style='padding: 6px;'>
Y_SCR
</td>
<td style='padding: 6px;'>
level_00
</td>
<td style='padding: 6px;'>
SUMMARY.SCR
</td>
<td style='padding: 6px;'>{{ stream_id }}</td>
{% if 'file_pattern' in archive_info[stream_id] %}
<td style='padding: 6px;'>{{ archive_info[stream_id]['file_pattern'] }}</td>
{% else %}
<td style='padding: 6px;'>N/A</td>
{% endif %}
{% if 'description' in archive_info[stream_id] %}
<td style='padding: 6px;'>{{ archive_info[stream_id]['description'] }}</td>
{% else %}
<td style='padding: 6px;'></td>
{% endif %}
</tr>
{% endfor %}
</table>
<h3>
Versions:
</h3>
<p>
Currently, there is only one version 'version_00' and it has been used from 06/08/2016-present
</p>
<h2>
ARM Product Levels:
</h2>
<p>
Currently, there is only one product level available, level_00 (raw data - primary raw data stream collected directly from instrument).
The product levels follow ARM's formatting and naming protocols. More information can be found
Product levels are based on the ARM data levels. More information can be found
<a href='https://www.arm.gov/data/docs/plan'>here.</a>
</p>
</div>
......
metobsapi/util/__init__.py
+ 3
− 3
View file @ a4cb0a38
......@@ -30,13 +30,13 @@ def create_fake_archive(archive_info, root=FAKE_ARCHIVE_PATH, datetimes=None):
for site, inst_info in archive_info.items():
os.makedirs(site, exist_ok=True)
os.chdir(site)
for inst, level_info in inst_info.items():
for inst, inst_info in inst_info['instruments'].items():
os.makedirs(inst, exist_ok=True)
os.chdir(inst)
for level_name, level_info in level_info.items():
for level_name, level_info in inst_info['levels'].items():
os.makedirs(level_name, exist_ok=True)
os.chdir(level_name)
for version_name in level_info['version']:
for version_name in level_info['versions']:
os.makedirs(version_name, exist_ok=True)
os.chdir(version_name)
for dt in datetimes:
......
metobsapi/util/file_responses.py
+ 71
− 59
View file @ a4cb0a38
......@@ -23,70 +23,82 @@ FREQUENCY_DIR_FMT = {
# TODO: Load from config file
ARCHIVE_INFO = {
'aoss': {
'tower': {
'level_00': {
'version': ('version_00',),
'products': {
'ascii': {
'frequency': ProductFrequency.DAILY_FILE,
'pattern': 'rig_tower.%Y-%m-%d.ascii',
'display_name': 'AOSS',
'instruments': {
'tower': {
'display_name': 'Tower',
'levels': {
'level_00': {
'versions': ('version_00',),
'products': {
'ascii': {
'frequency': ProductFrequency.DAILY_FILE,
'pattern': 'rig_tower.%Y-%m-%d.ascii',
},
},
},
},
},
'level_b1': {
'version': ('version_00',),
'products': {
'nc-1mo-1d': {
'frequency': ProductFrequency.MONTHLY_DIR,
'pattern': 'aoss_tower.%Y-%m.nc',
},
'nc-1d-1m': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': 'aoss_tower.%Y-%m-%d.nc',
'level_b1': {
'versions': ('version_00',),
'products': {
'nc-1mo-1d': {
'frequency': ProductFrequency.MONTHLY_DIR,
'pattern': 'aoss_tower.%Y-%m.nc',
},
'nc-1d-1m': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': 'aoss_tower.%Y-%m-%d.nc',
},
},
},
},
},
},
'aeri': {
'level_00': {
'version': ('version_00',),
'products': {
'par': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': '%y%m%d.par',
'version': ('version_00',),
},
'qc': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': '%y%m%d.qc',
'version': ('version_00',),
},
'sum': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': '%y%m%d.sum',
'version': ('version_00',),
},
'scr-aesitter': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': 'AESITTER.SCR',
'version': ('version_00',),
},
'scr-radiance': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': 'RADIANCE.SCR',
'version': ('version_00',),
},
'scr-summary': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': 'SUMMARY.SCR',
'version': ('version_00',),
'aeri': {
'display_name': 'AERI',
'levels': {
'level_00': {
'versions': ('version_00',),
'products': {
'par': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': '%y%m%d.par',
'version': ('version_00',),
},
'qc': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': '%y%m%d.qc',
'version': ('version_00',),
},
'sum': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': '%y%m%d.sum',
'version': ('version_00',),
},
'scr-aesitter': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': 'AESITTER.SCR',
'version': ('version_00',),
},
'scr-radiance': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': 'RADIANCE.SCR',
'version': ('version_00',),
},
'scr-summary': {
'frequency': ProductFrequency.DAILY_DIR,
'pattern': 'SUMMARY.SCR',
'version': ('version_00',),
},
},
},
},
},
},
},
'mendota': {
'display_name': 'Mendota',
'instruments': {
},
},
}
# Add the other AERI file types
......@@ -107,7 +119,7 @@ for file_suffix in ('B1.CXS',
'frequency': ProductFrequency.DAILY_DIR,
'pattern': '%y%m%d{}'.format(file_suffix),
}
ARCHIVE_INFO['aoss']['aeri']['level_00'][product_id] = nfo
ARCHIVE_INFO['aoss']['instruments']['aeri']['levels']['level_00'][product_id] = nfo
# Create stream_id -> stream_info mapping
ARCHIVE_STREAMS = {}
......@@ -115,11 +127,11 @@ ARCHIVE_STREAMS = {}
INSTRUMENT_STREAMS = defaultdict(list)
stream_id_fmt = "{site}.{inst}.{product}.{level}.{version}"
for site, site_info in ARCHIVE_INFO.items():
for inst, inst_info in site_info.items():
for inst, inst_info in site_info['instruments'].items():
inst_name = site + '.' + inst
for level, level_info in inst_info.items():
for level, level_info in inst_info['levels'].items():
all_products_recent = []
for version in level_info['version']:
for version in level_info['versions']:
all_products = []
for product_id, pattern_info in level_info['products'].items():
stream_id = stream_id_fmt.format(
......@@ -150,7 +162,7 @@ for site, site_info in ARCHIVE_INFO.items():
INSTRUMENT_STREAMS[inst_name].append(stream_id)
# Special 'most recent' version stream_id
if version == level_info['version'][0]:
if version == level_info['versions'][0]:
recent_stream_id = stream_id_fmt.format(
site=site,
inst=inst,
......@@ -170,7 +182,7 @@ for site, site_info in ARCHIVE_INFO.items():
)
ARCHIVE_STREAMS[all_patterns_id] = all_products
INSTRUMENT_STREAMS[inst_name].append(all_patterns_id)
if version == level_info['version'][0]:
if version == level_info['versions'][0]:
all_patterns_recent_id = all_patterns_id.replace(
version.replace('version_', 'v'), '*')
ARCHIVE_STREAMS[all_patterns_recent_id] = all_products
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment