Update hierarchicals
rzinke opened this issue · comments
Hi again. I am trying to set the fit_plane option to true for two interferograms (asc/dsc) for the July 2020 Xizang earthquake. Here I use only geodetic data. I initialize the problem and compute the greens functions.
I then use beat update Xizang --parameters=hierarchicals,hypers to update the parameters, with fit_plane = true. When I run beat sample Xizang --hypers I get a message that says:
(pyrocko) [rzinke@mpzinke-arch BeatTest6]$ beat sample Xizang --hypers
Duplicate key in file '/home/rzinke/.config/matplotlib/matplotlibrc', line 44 ('backend: Qt4Cairo')
config - INFO All hierarchicals ok!
config - INFO All hyperparameters ok!
config - INFO All priors ok!
models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ...
models - INFO ---------------------
geodetic - INFO Number of geodetic datasets: 2
geodetic - INFO Number of geodetic data points: 1111
geodetic - WARNING Covariance estimation not implemented (yet)! Using imported covariances!
geodetic - INFO Fit residual ramp selected!
models - INFO ... Building Hyper model ...
geodetic - INFO Estimating ramp for each dataset...
Traceback (most recent call last):
File "/home/rzinke/miniconda3/envs/pyrocko/bin/beat", line 33, in <module>
sys.exit(load_entry_point('beat==1.0', 'console_scripts', 'beat')())
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 1688, in main
globals()['command_' + command](args)
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 770, in command_sample
project_dir, options.mode, options.hypers)
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/problems.py", line 842, in load_model
problem.built_hyper_model()
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/problems.py", line 261, in built_hyper_model
self.init_hierarchicals()
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/problems.py", line 546, in init_hierarchicals
composite.init_hierarchicals(self.config.problem_config)
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/geodetic.py", line 242, in init_hierarchicals
'Plane corrections enabled, but they are'
beat.models.base.ConfigInconsistentError: Plane corrections enabled, but they are not defined in the problem configuration! (hierarchicals)
Please run: "beat update <project_dir> --parameters="hierarchicals"
I re-run beat update Xizang --parameters=hierarchicals,hypers to be sure. But I still get the same error message.
I checked the config_geometry.yaml file and the hierarchicals are definitely there:
hierarchicals:
Tr121dsc_ct03_e002n05m001M01_kite_scene_offset: !beat.heart.Parameter
name: Tr121dsc_ct03_e002n05m001M01_kite_scene_offset
form: Uniform
lower:
- -0.05
upper:
- 0.05
testvalue:
- 0.0
Tr121dsc_ct03_e002n05m001M01_kite_scene_ramp: !beat.heart.Parameter
name: Tr121dsc_ct03_e002n05m001M01_kite_scene_ramp
form: Uniform
lower:
- -0.005
- -0.005
upper:
- 0.005
- 0.005
testvalue:
- 0.0
- 0.0
Tr12asc_ct03_e002n05m001M01_kite_scene_offset: !beat.heart.Parameter
name: Tr12asc_ct03_e002n05m001M01_kite_scene_offset
form: Uniform
lower:
- -0.05
upper:
- 0.05
testvalue:
- 0.0
Tr12asc_ct03_e002n05m001M01_kite_scene_ramp: !beat.heart.Parameter
name: Tr12asc_ct03_e002n05m001M01_kite_scene_ramp
form: Uniform
lower:
- -0.005
- -0.005
upper:
- 0.005
- 0.005
testvalue:
- 0.0
- 0.0
Not sure what to do.
Can you please check your kite scenes? I think their filenames do include the path and not only the filename.
Can you please remove the path in front of the filename?
You can do this by opening them in the spool gui. Then clicking on the filename and possible edit them there. Then save...
Once you did so you need to reimport the data with beat import.
Please let me know if that is indeed the issue.
The two kite scenes are only the filenames (not prepended by paths).
The scene names are long:
geodetic_config: !beat.GeodeticConfig
datadir: /home/rzinke/Titan/Projects/Xizang/KiteQuadtree/
names:
- Tr12asc_ct03_e002n05m001M01_kite_scene
- Tr121dsc_ct03_e002n05m001M01_kite_scene
Should I try re-importing anyway?
Here is my config_geometry.yaml file (attached)
config_geometry.txt
I am not talking about the beat config. You need to open the KITE scenes!
Can you please check your kite scenes? I think their filenames do include the path and not only the filename.
Can you please remove the path in front of the filename?
You can do this by opening them in the spool gui. Then clicking on the filename and possible edit them there. Then save...
Once you did so you need to reimport the data with beat import.
Please let me know if that is indeed the issue.
I opened the scenes using spool, but did not see the filename option. Is it possible to update the filename option in the .yml file?
Thanks
Sure just do it there, open it with vi or whatever texteditor and change it there. It must only contain the filename not the full path.
I updated the yml files of my Kite scenes. For example, here are the first several lines of Tr121dsc_ct03_e002n05m001M01_kite_scene.yml:
(pyrocko) [rzinke@mpzinke-arch BeatTest2]$ more ../KiteQuadtree/Tr121dsc_ct03_e002n05m001M01_kite_scene.yml
%YAML 1.1
# kite.Scene YAML Config
--- !kite.scene.SceneConfig
meta: !kite.scene.Meta
scene_title: Unnamed Scene
scene_id: None
satellite_name: Undefined Mission
orbital_node: Undefined
time_master: 2016-12-07 13:09:21.930574
time_slave: 2016-12-20 13:08:45.482000
filename: filt_topophase_2stage.unw.geo
frame: !kite.scene.FrameConfig
llLat: 32.45916666666667
llLon: 86.20277777777778
dN: 0.0002777777777777778
dE: 0.0002777777777777778
spacing: degree
I started a new project, ran beat import Xizang and it recomputed the covariances. I ran beat update Xizang --parameters='hierarchicals,hypers'.
Once again, however, beat sample Xizang --hypers produces the error:
(pyrocko) [rzinke@mpzinke-arch BeatTest2]$ beat sample Xizang --hypers
config - INFO All hierarchicals ok!
config - INFO All hyperparameters ok!
config - INFO All priors ok!
models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ...
models - INFO ---------------------
geodetic - INFO Number of geodetic datasets: 3
geodetic - INFO Number of geodetic data points: 1455
geodetic - WARNING Covariance estimation not implemented (yet)! Using imported covariances!
geodetic - INFO Fit residual ramp selected!
models - INFO ... Building Hyper model ...
geodetic - INFO Estimating ramp for each dataset...
Traceback (most recent call last):
File "/home/rzinke/miniconda3/envs/pyrocko/bin/beat", line 33, in <module>
sys.exit(load_entry_point('beat==1.0', 'console_scripts', 'beat')())
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 1688, in main
globals()['command_' + command](args)
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 770, in command_sample
project_dir, options.mode, options.hypers)
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/problems.py", line 842, in load_model
problem.built_hyper_model()
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/problems.py", line 261, in built_hyper_model
self.init_hierarchicals()
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/problems.py", line 546, in init_hierarchicals
composite.init_hierarchicals(self.config.problem_config)
File "/home/rzinke/miniconda3/envs/pyrocko/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/geodetic.py", line 242, in init_hierarchicals
'Plane corrections enabled, but they are'
beat.models.base.ConfigInconsistentError: Plane corrections enabled, but they are not defined in the problem configuration! (hierarchicals)
Please run: "beat update <project_dir> --parameters="hierarchicals"
So according to your yaml file above your scene name is: Tr121dsc_ct03_e002n05m001M01_kite_scene
So please put that under filename where you have now: filt_topophase_2stage.unw.geo
Then you need to re-import the geodetic data through beat import.
Thanks for the suggestion, @hvasbath. I updated the ymls files to look like this, e.g.,
(pyrocko) [rzinke@mpzinke-arch BeatTest2]$ more ../KiteQuadtree/Tr121dsc_ct03_e002n05m001M01_kite_scene.yml
%YAML 1.1
# kite.Scene YAML Config
--- !kite.scene.SceneConfig
meta: !kite.scene.Meta
scene_title: Unnamed Scene
scene_id: None
satellite_name: Undefined Mission
orbital_node: Undefined
time_master: 2016-12-07 13:09:21.930574
time_slave: 2016-12-20 13:08:45.482000
filename: Tr121dsc_ct03_e002n05m001M01_kite_scene
frame: !kite.scene.FrameConfig
llLat: 32.45916666666667
llLon: 86.20277777777778
dN: 0.0002777777777777778
dE: 0.0002777777777777778
spacing: degree
quadtree: !kite.quadtree.QuadtreeConfig
correction: median
epsilon: 0.02
nan_allowed: 0.5
tile_size_min: 0.01
tile_size_max: 0.1
covariance: !kite.covariance.CovarianceConfig
model_function: exponential
sampling_method: spatial
spatial_bins: 75
spatial_pairs: 200000
adaptive_subsampling: true
aps: !kite.aps.APSConfig
applied: false
gacos: !kite.gacos.GACOSConfig
applied: false
polygon_mask: !kite.scene_mask.PolygonMaskConfig
applied: true
deramp: !kite.deramp.DerampConfig
applied: false
I re-imported using beat import Xizang --force, and reran all steps. However, beat sample Xizang --hypers produces the same error message.
You are on the master branch?
Did once run beat update with --parameters=hierarchicals?
I have no clue sorry. You can try using a print statement in beat.models.geodetic line 231:
print(hierarchical_name, hierarchicals)
Please be aware indentation.
Need to rerun python3 setup.py install after change.
The hierarchical name needs to be inside hierarchicals ...
Another thing you may try without print is using simpler filenames.
I added verbosity to the error message. Can you please update your master and rerun- if you havent bothered with the print statements yet. @rzinke
Hi @rzinke Did you solve this problem?
HI @hvasbath I meet the same problems as @rzinke . The Scnario 3 and Scnario 4 works out beautifully. But when I used beat for my own data, the error occurs when I run beat sample BalochistanEQ --hypers. I tried all possible solutions you referred above but the problem is still there......
The error message is:
[lxrtest@dqcentos7insar BEAT]$ beat sample BalochistanEQ --hypers
config - INFO All hierarchicals ok!
config - INFO All hyperparameters ok!
config - INFO All priors ok!
models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ...
models - INFO ---------------------
geodetic - INFO Number of geodetic datasets: 2
geodetic - INFO Number of geodetic data points: 919
geodetic - WARNING Covariance estimation not implemented (yet)! Using imported covariances!
geodetic - INFO Fit residual ramp selected!
models - INFO ... Building Hyper model ...
geodetic - INFO Estimating ramp for each dataset...
Traceback (most recent call last):
File "/home/lxrtest/wclsolid/pyenv/versions/miniconda3-4.7.12/bin/beat", line 11, in <module>
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lxrtest/wclsolid/pyenv/versions/miniconda3-4.7.12/lib/python3.7/site-packages/beat-1.0-py3.7-linux-x86_64.egg/beat/apps/beat.py", line 1688, in main
globals()['command_' + command](args)
File "/home/lxrtest/wclsolid/pyenv/versions/miniconda3-4.7.12/lib/python3.7/site-packages/beat-1.0-py3.7-linux-x86_64.egg/beat/apps/beat.py", line 770, in command_sample
project_dir, options.mode, options.hypers)
File "/home/lxrtest/wclsolid/pyenv/versions/miniconda3-4.7.12/lib/python3.7/site-packages/beat-1.0-py3.7-linux-x86_64.egg/beat/models/problems.py", line 842, in load_model
problem.built_hyper_model()
File "/home/lxrtest/wclsolid/pyenv/versions/miniconda3-4.7.12/lib/python3.7/site-packages/beat-1.0-py3.7-linux-x86_64.egg/beat/models/problems.py", line 261, in built_hyper_model
self.init_hierarchicals()
File "/home/lxrtest/wclsolid/pyenv/versions/miniconda3-4.7.12/lib/python3.7/site-packages/beat-1.0-py3.7-linux-x86_64.egg/beat/models/problems.py", line 546, in init_hierarchicals
composite.init_hierarchicals(self.config.problem_config)
File "/home/lxrtest/wclsolid/pyenv/versions/miniconda3-4.7.12/lib/python3.7/site-packages/beat-1.0-py3.7-linux-x86_64.egg/beat/models/geodetic.py", line 249, in init_hierarchicals
hierarchical_name, hierarchical_keys))
beat.models.base.ConfigInconsistentError: Plane corrections enabled, but they are not defined in the problem configuration! (hierarchicals). Looking for: /data/lxrtest/insarlab/BEAT/Balochistan_data/Sen201410_201712asc_ramp
Got: "Sen201410_201712asc_offset", "Sen201410_201712asc_ramp", "Sen201410_201712des_offset", "Sen201410_201712des_ramp"
Please run: "beat update <project_dir> --parameters="hierarchicals"
And the kite yml file is
[lxrtest@dqcentos7insar Balochistan_data]$ cat Sen201410_201712asc.yml
%YAML 1.1
# kite.Scene YAML Config
--- !kite.scene.SceneConfig
meta: !kite.scene.Meta
scene_title: Unnamed Scene
scene_id: None
satellite_name: Undefined Mission
wavelength: 0.05546576
orbital_node: Undefined
time_master: 2016-12-07 13:09:21.930574
time_slave: 2016-12-20 13:08:45.482000
filename: Sen201410_201712asc
frame: !kite.scene.FrameConfig
llLat: 25.117615890502933
llLon: 62.30580084228516
dN: 0.002
dE: 0.002
spacing: degree
quadtree: !kite.quadtree.QuadtreeConfig
correction: median
epsilon: 0.013
nan_allowed: 0.8
tile_size_min: 0.04
tile_size_max: 0.6
covariance: !kite.covariance.CovarianceConfig
noise_coord:
- 0.0
- 2.048
- 1.024
- 1.024
model_coefficients:
- 4.4177838594810496e-05
- 9942.99568310541
model_function: exponential
sampling_method: spatial
spatial_bins: 75
spatial_pairs: 200000
variance: 4.244293402482782e-05
adaptive_subsampling: true
aps: !kite.aps.APSConfig
applied: false
gacos: !kite.gacos.GACOSConfig
applied: false
polygon_mask: !kite.scene_mask.PolygonMaskConfig
applied: true
deramp: !kite.deramp.DerampConfig
applied: false
And my config_geodetic.yaml file is:
[lxrtest@dqcentos7insar BalochistanEQ]$ cat config_geometry.yaml
--- !beat.BEATconfig
name: BalochistanEQ
date: '2013-09-24'
event: !pf.Event
lat: 26.7
lon: 65.04
time: 2013-09-24 11:30:08.410000
depth: 12000.0
name: '201309241129A'
magnitude: 7.802455896833244
region: 'PAKISTAN'
catalog: 'gCMT'
moment_tensor: !pf.MomentTensor
mnn: -3.88e+20
mee: 3.36e+20
mdd: 5.2e+19
mne: -2.5e+18
mnd: 3.72e+20
med: 2.2400000000000003e+20
strike1: 129.5411241357749
dip1: 87.23664668920499
rake1: 128.95950855038447
strike2: 222.95315273175524
dip2: 39.04182922930022
rake2: 4.38965037006906
moment: 5.671315984848667e+20
magnitude: 7.802455896833244
duration: 37.2
project_dir: /data/lxrtest/insarlab/BEAT/BalochistanEQ
problem_config: !beat.ProblemConfig
mode: geometry
source_type: RectangularSource
stf_type: HalfSinusoid
decimation_factors:
geodetic: 4
n_sources: 1
datatypes:
- geodetic
hyperparameters:
h_SAR: !beat.heart.Parameter
name: h_SAR
form: Uniform
lower:
- -20.0
upper:
- 20.0
testvalue:
- 0.0
priors:
depth: !beat.heart.Parameter
name: depth
form: Uniform
lower:
- 0.0
upper:
- 1.0
testvalue:
- 0.2
dip: !beat.heart.Parameter
name: dip
form: Uniform
lower:
- 30.0
upper:
- 70.0
testvalue:
- 40.0
east_shift: !beat.heart.Parameter
name: east_shift
form: Uniform
lower:
- -20.0
upper:
- 20.0
testvalue:
- -8.0
length: !beat.heart.Parameter
name: length
form: Uniform
lower:
- 200.0
upper:
- 250.0
testvalue:
- 230.0
north_shift: !beat.heart.Parameter
name: north_shift
form: Uniform
lower:
- -20.0
upper:
- 20.0
testvalue:
- -8.0
rake: !beat.heart.Parameter
name: rake
form: Uniform
lower:
- 0.0
upper:
- 10.0
testvalue:
- 4.0
slip: !beat.heart.Parameter
name: slip
form: Uniform
lower:
- 1.0
upper:
- 8.0
testvalue:
- 2.5
strike: !beat.heart.Parameter
name: strike
form: Uniform
lower:
- 180.0
upper:
- 270.0
testvalue:
- 230.0
width: !beat.heart.Parameter
name: width
form: Uniform
lower:
- 25.0
upper:
- 35.0
testvalue:
- 32.0
hierarchicals:
Sen201410_201712asc_offset: !beat.heart.Parameter
name: Sen201410_201712asc_offset
form: Uniform
lower:
- -0.05
upper:
- 0.05
testvalue:
- 0.0
Sen201410_201712asc_ramp: !beat.heart.Parameter
name: Sen201410_201712asc_ramp
form: Uniform
lower:
- -0.005
- -0.005
upper:
- 0.005
- 0.005
testvalue:
- 0.0
- 0.0
Sen201410_201712des_offset: !beat.heart.Parameter
name: Sen201410_201712des_offset
form: Uniform
lower:
- -0.05
upper:
- 0.05
testvalue:
- 0.0
Sen201410_201712des_ramp: !beat.heart.Parameter
name: Sen201410_201712des_ramp
form: Uniform
lower:
- -0.005
- -0.005
upper:
- 0.005
- 0.005
testvalue:
- 0.0
- 0.0
geodetic_config: !beat.GeodeticConfig
datadir: /data/lxrtest/insarlab/BEAT/Balochistan_data
names:
- Sen201410_201712asc
- Sen201410_201712des
types:
- SAR
calc_data_cov: true
interpolation: multilinear
fit_plane: true
dataset_specific_residual_noise_estimation: true
gf_config: !beat.GeodeticGFConfig
store_superdir: /data/lxrtest/insarlab/GF/Balochistan
reference_model_idx: 0
n_variations:
- 0
- 1
earth_model_name: ak135-f-average.m
nworkers: 50
use_crust2: true
replace_water: true
source_depth_min: 0.0
source_depth_max: 27.5
source_depth_spacing: 1.0
source_distance_radius: 160.0
source_distance_spacing: 1.0
error_depth: 0.1
error_velocities: 0.1
depth_limit_variation: 600.0
code: psgrn
sample_rate: 1.1574074074074073e-05
sampling_interval: 1.0
medium_depth_spacing: 1.0
medium_distance_spacing: 10.0
sampler_config: !beat.SamplerConfig
name: SMC
backend: csv
progressbar: true
buffer_size: 5000
buffer_thinning: 1
parameters: !beat.SMCConfig
tune_interval: 50
check_bnd: true
rm_flag: false
n_jobs: 1
n_steps: 100
n_chains: 1000
coef_variation: 1.0
stage: 0
proposal_dist: MultivariateNormal
update_covariances: false
hyper_sampler_config: !beat.SamplerConfig
name: Metropolis
backend: csv
progressbar: true
buffer_size: 5000
buffer_thinning: 1
parameters: !beat.MetropolisConfig
tune_interval: 50
proposal_dist: Normal
check_bnd: true
rm_flag: false
n_jobs: 1
n_steps: 25000
n_chains: 20
thin: 2
burn: 0.5
Could you please give me any other suggestions?
Thank you!
Do you mind updating beat to the development version? We are close to a new release where such stuff should be fixed, but it will be a huge pain to try to debug the master branch.
In the beat directory:
git fetch origin v1.1_dev_rc
git checkout v1.1_dev_rc
git pull origin v1.1_dev_rc
then install:
python3 setup.py install (or with sudo depending on your settings and system)
Then your config files need to be updated- please back them up before.
https://hvasbath.github.io/beat/getting_started/update_config.html
New for you would be the corrections configs- I had no time to write a tutorial about these, for you relevant would be only the ramp_correction, you need to enable them and insert your dataset names again, example:
corrections_config: !beat.GeodeticCorrectionsConfig
euler_poles:
- !beat.EulerPoleConfig
dataset_names: [velocity_211219.txt, T138_mask_fine_v1.yml, T36a_new_mask_v1.yml,
T131a_mask_v1.yml]
enabled: false
station_blacklist: [KR01_MGA, KR02_MGA, KR03_MGA, KR04_MGA, KR10_MGA, KR11_MGA]
ramp: !beat.RampConfig
dataset_names: [T138_mask_fine_v1.yml, T36a_new_mask_v1.yml, T131a_mask_v1.yml]
enabled: true
The development version also adds support for multisegmented faults.
In case further questions come up- please do not hesitate to contact me. Its time to prepare the tutorials for the release ...