beat.sampler.distributed.MPIError: #54
Comments
|
Ah ok saw that you posted everything. The workaround for now would be to downgrade. Does it work using the --oversubscribe option ergo running any of the mpiexec commands above for example with -n 9 |
|
Hi. I put n_chains= 11 and ran beat sample Laquila_dc; as it was expected I got the error: distributed - INFO MPI is installed! However running in the temporary directory: mpiexec --use-hwthread-cpus -n 12 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc I did not get any error and mpiexec started to execute the process. When n_chains=12; mpiexec did not work. Thanks. |
|
Sure as this will try to create 13 processes which computer doesn't have threads. |
|
Hi. I ran n_chains=12 and n_chains=16: mpiexec --oversubscribe -n 13 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc mpiexec --oversubscribe -n 17 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc I did not get any error and mpiexec started to execute the process. |
|
Great so thats the flag then we have to include in the execution string. We will have to check if it is available across versions- I will do that. |
|
This seems to have deeper roots connected to issues with MPI distributions. Collecting here a bunch of relevant threads to further investigate: |
|
A fast workaround for that issue could be to ensure to use older MPI versions (<3): Also updated webpage install instructions with that... |
|
need to cleanup master. .. |
|
OpenMPI version 3.3.2 works with mpi4py |
|
Users that used anaconda reported that this issue could be fixed installing mpi4py through conda-forge like this: |
Hi all,
1) Running the second example with n_chains=8, I got:
(beat) lab@lab:~$ beat sample Laquila_dc
config - INFO All hierarchicals ok!
config - INFO All hyperparameters ok!
config - INFO All priors ok!
models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ...
models - INFO ---------------------
heart - INFO Consistent number of datasets and targets in any_P_0 wavemap!
heart - INFO Number of seismic datasets for wavemap: any_P_0: 35
models - INFO ... Building model ...
models - INFO Optimization for 35 hyperparameters in total!
seismic - INFO Seismic optimization on:
depth, dip, duration, east_shift, magnitude, north_shift, rake, strike, time
seismic - INFO Retrieving seismic data-covariances with structure "variance" for any_P_0 ...
seismic - INFO Preparing data of "any_P_0" for optimization
seismic - INFO Initializing synthesizer for "any_P_0"
models - INFO Model building was successful!
models - INFO Using "bin" backend to store samples!
models - INFO ... Initiate Metropolis for Parallel Tempering...
proposal_distribution: Normal, tune_interval=10, n_chains=8
models.base - INFO ... Starting Parallel Tempering ...
distributed - INFO MPI is installed!
distributed - INFO Detected python3
distributed - INFO Done initialising mpi runner
distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py
Traceback (most recent call last):
File "/home/lab/anaconda3/envs/beat/bin/beat", line 11, in
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 1686, in main
globals()'command_' + command
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 775, in command_sample
sample(step, problem)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/base.py", line 196, in sample
record_worker_chains=pa.record_worker_chains)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/pt.py", line 836, in pt_sample
project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 226, in run_mpi_sampler
samplerdir, n_jobs=n_jobs, loglevel=loglevel, project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 158, in run
in the directory %s'''.lstrip() % (program, self.tempdir))
beat.sampler.distributed.MPIError: mpiexec has been invoked as "mpiexec"
in the directory /tmp/mpiexec-56qm7lyu
distributed - WARNING not removing temporary directory: /tmp/mpiexec-56qm7lyu
------------------------------------------------------------*-
2. Running again in --loglevel=debug:
(beat) lab@lab:~$ beat sample Laquila_dc
config - INFO All hierarchicals ok!
config - INFO All hyperparameters ok!
config - INFO All priors ok!
models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ...
models - INFO ---------------------
heart - INFO Consistent number of datasets and targets in any_P_0 wavemap!
heart - INFO Number of seismic datasets for wavemap: any_P_0: 35
models - INFO ... Building model ...
models - INFO Optimization for 35 hyperparameters in total!
seismic - INFO Seismic optimization on:
depth, dip, duration, east_shift, magnitude, north_shift, rake, strike, time
seismic - INFO Retrieving seismic data-covariances with structure "variance" for any_P_0 ...
seismic - INFO Preparing data of "any_P_0" for optimization
seismic - INFO Initializing synthesizer for "any_P_0"
models - INFO Model building was successful!
models - INFO Using "bin" backend to store samples!
models - INFO ... Initiate Metropolis for Parallel Tempering...
proposal_distribution: Normal, tune_interval=10, n_chains=8
models.base - INFO ... Starting Parallel Tempering ...
distributed - INFO MPI is installed!
distributed - INFO Detected python3
distributed - INFO Done initialising mpi runner
distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py
Traceback (most recent call last):
File "/home/lab/anaconda3/envs/beat/bin/beat", line 11, in
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 1686, in main
globals()'command_' + command
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 775, in command_sample
sample(step, problem)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/base.py", line 196, in sample
record_worker_chains=pa.record_worker_chains)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/pt.py", line 836, in pt_sample
project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 226, in run_mpi_sampler
samplerdir, n_jobs=n_jobs, loglevel=loglevel, project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 158, in run
in the directory %s'''.lstrip() % (program, self.tempdir))
beat.sampler.distributed.MPIError: mpiexec has been invoked as "mpiexec"
in the directory /tmp/mpiexec-56qm7lyu
distributed - WARNING not removing temporary directory: /tmp/mpiexec-56qm7lyu
(beat) lab@lab:~$ beat sample Laquila_dc --loglevel=debug
config - INFO All hierarchicals ok!
config - INFO All hyperparameters ok!
config - INFO All priors ok!
models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ...
models - INFO ---------------------
seismic - DEBUG Setting up seismic structure ...
utility - DEBUG Valid distance range: [30.000000, 90.000000]!
utility - DEBUG Distance of station TSUM: 61.349855 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station RCBR: 65.567308 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station RES: 54.143590 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KMI: 73.109289 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station SSPA: 64.946397 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station BTDF: 89.437223 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station ANWB: 67.439618 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station MBO: 38.167185 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station TEIG: 84.816714 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KBL: 43.633450 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station MSEY: 60.417579 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station DBIC: 38.986606 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station HIA: 68.244425 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station YAK: 64.054778 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station QIZ: 82.062195 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KBS: 36.698309 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station LSZ: 58.931762 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station TLY: 58.456908 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station INCN: 79.971426 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station DWPF: 74.925103 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station WMQ: 52.564826 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station AAK: 44.192670 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station XAN: 71.681441 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KMBO: 48.335702 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KURK: 44.334011 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station ADYE: 36.168619 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station SDV: 79.664513 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station ARU: 31.995457 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station SAML: 86.199701 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station ALE: 46.405043 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station OTAV: 91.209244 [deg]
utility - DEBUG Outside defined distance range!
utility - DEBUG Distance of station ABKT: 34.203018 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station AKTO: 31.451507 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station BJT: 72.303754 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station SSE: 81.349799 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station CHTO: 74.397519 [deg]
utility - DEBUG Inside defined distance range!
heart - INFO Consistent number of datasets and targets in any_P_0 wavemap!
heart - INFO Number of seismic datasets for wavemap: any_P_0: 35
models - INFO ... Building model ...
models - DEBUG Optimization for 1 sources
models - INFO Optimization for 35 hyperparameters in total!
seismic - INFO Seismic optimization on:
depth, dip, duration, east_shift, magnitude, north_shift, rake, strike, time
seismic - INFO Retrieving seismic data-covariances with structure "variance" for any_P_0 ...
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 61.349855 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 65.567308 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 54.143590 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 73.109289 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 64.946397 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 89.437223 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 67.439618 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 38.167185 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 84.816714 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 43.633450 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 60.417579 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 38.986606 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 68.244425 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 64.054778 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 82.062195 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 36.698309 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 58.931762 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 58.456908 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 79.971426 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 74.925103 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 52.564826 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 44.192670 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 71.681441 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 48.335702 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 44.334011 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 36.168619 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 79.664513 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 31.995457 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 86.199701 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 46.405043 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 34.203018 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 31.451507 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 72.303754 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 81.349799 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 74.397519 [deg]
seismic - INFO Preparing data of "any_P_0" for optimization
heart - DEBUG Arrival time for wavename "any_P" distance 61.349855 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 65.567308 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 54.143590 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 73.109289 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 64.946397 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 89.437223 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 67.439618 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 38.167185 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 84.816714 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 43.633450 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 60.417579 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 38.986606 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 68.244425 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 64.054778 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 82.062195 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 36.698309 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 58.931762 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 58.456908 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 79.971426 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 74.925103 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 52.564826 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 44.192670 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 71.681441 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 48.335702 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 44.334011 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 36.168619 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 79.664513 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 31.995457 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 86.199701 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 46.405043 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 34.203018 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 31.451507 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 72.303754 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 81.349799 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 74.397519 [deg]
heart - DEBUG Pre-processing data ...
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1641
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982169.0
b: 1238982174.0
c: 1238982224.0
d: 1238982239.0
heart - DEBUG trace: Trace (IU, TSUM, filt, Z)
timerange: 2009-04-06 01:32:44 - 2009-04-06 02:27:24
delta t: 2
heart - DEBUG chopped trace: Trace (IU, TSUM, filt, Z)
timerange: 2009-04-06 01:42:54 - 2009-04-06 01:43:42
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1716
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982197.0
b: 1238982202.0
c: 1238982252.0
d: 1238982267.0
heart - DEBUG trace: Trace (IU, RCBR, filt, Z)
timerange: 2009-04-06 01:33:12 - 2009-04-06 02:30:22
delta t: 2
heart - DEBUG chopped trace: Trace (IU, RCBR, filt, Z)
timerange: 2009-04-06 01:43:22 - 2009-04-06 01:44:10
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1508
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982119.0
b: 1238982124.0
c: 1238982174.0
d: 1238982189.0
heart - DEBUG trace: Trace (CN, RES, filt, Z)
timerange: 2009-04-06 01:31:52 - 2009-04-06 02:22:06
delta t: 2
heart - DEBUG chopped trace: Trace (CN, RES, filt, Z)
timerange: 2009-04-06 01:42:04 - 2009-04-06 01:42:52
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1853
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982245.0
b: 1238982250.0
c: 1238982300.0
d: 1238982315.0
heart - DEBUG trace: Trace (IC, KMI, filt, Z)
timerange: 2009-04-06 01:34:00 - 2009-04-06 02:35:44
delta t: 2
heart - DEBUG chopped trace: Trace (IC, KMI, filt, Z)
timerange: 2009-04-06 01:44:10 - 2009-04-06 01:44:58
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1702
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982193.0
b: 1238982198.0
c: 1238982248.0
d: 1238982263.0
heart - DEBUG trace: Trace (IU, SSPA, filt, Z)
timerange: 2009-04-06 01:33:08 - 2009-04-06 02:29:50
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SSPA, filt, Z)
timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:06
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2169
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982331.0
b: 1238982336.0
c: 1238982386.0
d: 1238982401.0
heart - DEBUG trace: Trace (MS, BTDF, filt, Z)
timerange: 2009-04-06 01:35:26 - 2009-04-06 02:47:42
delta t: 2
heart - DEBUG chopped trace: Trace (MS, BTDF, filt, Z)
timerange: 2009-04-06 01:45:36 - 2009-04-06 01:46:24
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1748
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982209.0
b: 1238982214.0
c: 1238982264.0
d: 1238982279.0
heart - DEBUG trace: Trace (CU, ANWB, filt, Z)
timerange: 2009-04-06 01:33:24 - 2009-04-06 02:31:38
delta t: 2
heart - DEBUG chopped trace: Trace (CU, ANWB, filt, Z)
timerange: 2009-04-06 01:43:34 - 2009-04-06 01:44:22
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1229
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981993.0
b: 1238981998.0
c: 1238982048.0
d: 1238982063.0
heart - DEBUG trace: Trace (G, MBO, filt, Z)
timerange: 2009-04-06 01:29:48 - 2009-04-06 02:10:44
delta t: 2
heart - DEBUG chopped trace: Trace (G, MBO, filt, Z)
timerange: 2009-04-06 01:39:58 - 2009-04-06 01:40:46
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2078
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982309.0
b: 1238982314.0
c: 1238982364.0
d: 1238982379.0
heart - DEBUG trace: Trace (IU, TEIG, filt, Z)
timerange: 2009-04-06 01:35:02 - 2009-04-06 02:44:16
delta t: 2
heart - DEBUG chopped trace: Trace (IU, TEIG, filt, Z)
timerange: 2009-04-06 01:45:14 - 2009-04-06 01:46:02
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1324
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982039.0
b: 1238982044.0
c: 1238982094.0
d: 1238982109.0
heart - DEBUG trace: Trace (IU, KBL, filt, Z)
timerange: 2009-04-06 01:30:32 - 2009-04-06 02:14:38
delta t: 2
heart - DEBUG chopped trace: Trace (IU, KBL, filt, Z)
timerange: 2009-04-06 01:40:44 - 2009-04-06 01:41:32
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1624
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982163.0
b: 1238982168.0
c: 1238982218.0
d: 1238982233.0
heart - DEBUG trace: Trace (II, MSEY, filt, Z)
timerange: 2009-04-06 01:32:38 - 2009-04-06 02:26:44
delta t: 2
heart - DEBUG chopped trace: Trace (II, MSEY, filt, Z)
timerange: 2009-04-06 01:42:48 - 2009-04-06 01:43:36
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1244
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982001.0
b: 1238982006.0
c: 1238982056.0
d: 1238982071.0
heart - DEBUG trace: Trace (GT, DBIC, filt, Z)
timerange: 2009-04-06 01:29:54 - 2009-04-06 02:11:20
delta t: 2
heart - DEBUG chopped trace: Trace (GT, DBIC, filt, Z)
timerange: 2009-04-06 01:40:06 - 2009-04-06 01:40:54
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1763
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982215.0
b: 1238982220.0
c: 1238982270.0
d: 1238982285.0
heart - DEBUG trace: Trace (IC, HIA, filt, Z)
timerange: 2009-04-06 01:33:30 - 2009-04-06 02:32:14
delta t: 2
heart - DEBUG chopped trace: Trace (IC, HIA, filt, Z)
timerange: 2009-04-06 01:43:40 - 2009-04-06 01:44:28
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1686
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982187.0
b: 1238982192.0
c: 1238982242.0
d: 1238982257.0
heart - DEBUG trace: Trace (IU, YAK, filt, Z)
timerange: 2009-04-06 01:33:02 - 2009-04-06 02:29:12
delta t: 2
heart - DEBUG chopped trace: Trace (IU, YAK, filt, Z)
timerange: 2009-04-06 01:43:12 - 2009-04-06 01:44:00
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2026
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982295.0
b: 1238982300.0
c: 1238982350.0
d: 1238982365.0
heart - DEBUG trace: Trace (IC, QIZ, filt, Z)
timerange: 2009-04-06 01:34:48 - 2009-04-06 02:42:18
delta t: 2
heart - DEBUG chopped trace: Trace (IC, QIZ, filt, Z)
timerange: 2009-04-06 01:45:00 - 2009-04-06 01:45:48
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1201
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981981.0
b: 1238981986.0
c: 1238982036.0
d: 1238982051.0
heart - DEBUG trace: Trace (GE, KBS, filt, Z)
timerange: 2009-04-06 01:29:34 - 2009-04-06 02:09:34
delta t: 2
heart - DEBUG chopped trace: Trace (GE, KBS, filt, Z)
timerange: 2009-04-06 01:39:46 - 2009-04-06 01:40:34
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1598
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982153.0
b: 1238982158.0
c: 1238982208.0
d: 1238982223.0
heart - DEBUG trace: Trace (IU, LSZ, filt, Z)
timerange: 2009-04-06 01:32:28 - 2009-04-06 02:25:42
delta t: 2
heart - DEBUG chopped trace: Trace (IU, LSZ, filt, Z)
timerange: 2009-04-06 01:42:38 - 2009-04-06 01:43:26
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1586
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982149.0
b: 1238982154.0
c: 1238982204.0
d: 1238982219.0
heart - DEBUG trace: Trace (II, TLY, filt, Z)
timerange: 2009-04-06 01:32:24 - 2009-04-06 02:25:14
delta t: 2
heart - DEBUG chopped trace: Trace (II, TLY, filt, Z)
timerange: 2009-04-06 01:42:34 - 2009-04-06 01:43:22
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1984
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982283.0
b: 1238982288.0
c: 1238982338.0
d: 1238982353.0
heart - DEBUG trace: Trace (IU, INCN, filt, Z)
timerange: 2009-04-06 01:34:38 - 2009-04-06 02:40:44
delta t: 2
heart - DEBUG chopped trace: Trace (IU, INCN, filt, Z)
timerange: 2009-04-06 01:44:48 - 2009-04-06 01:45:36
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1888
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982255.0
b: 1238982260.0
c: 1238982310.0
d: 1238982325.0
heart - DEBUG trace: Trace (IU, DWPF, filt, Z)
timerange: 2009-04-06 01:34:08 - 2009-04-06 02:37:02
delta t: 2
heart - DEBUG chopped trace: Trace (IU, DWPF, filt, Z)
timerange: 2009-04-06 01:44:20 - 2009-04-06 01:45:08
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1481
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982107.0
b: 1238982112.0
c: 1238982162.0
d: 1238982177.0
heart - DEBUG trace: Trace (IC, WMQ, filt, Z)
timerange: 2009-04-06 01:31:42 - 2009-04-06 02:21:02
delta t: 2
heart - DEBUG chopped trace: Trace (IC, WMQ, filt, Z)
timerange: 2009-04-06 01:41:52 - 2009-04-06 01:42:40
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1334
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982043.0
b: 1238982048.0
c: 1238982098.0
d: 1238982113.0
heart - DEBUG trace: Trace (II, AAK, filt, Z)
timerange: 2009-04-06 01:30:36 - 2009-04-06 02:15:02
delta t: 2
heart - DEBUG chopped trace: Trace (II, AAK, filt, Z)
timerange: 2009-04-06 01:40:48 - 2009-04-06 01:41:36
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1827
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982235.0
b: 1238982240.0
c: 1238982290.0
d: 1238982305.0
heart - DEBUG trace: Trace (IC, XAN, filt, Z)
timerange: 2009-04-06 01:33:50 - 2009-04-06 02:34:42
delta t: 2
heart - DEBUG chopped trace: Trace (IC, XAN, filt, Z)
timerange: 2009-04-06 01:44:00 - 2009-04-06 01:44:48
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1408
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982075.0
b: 1238982080.0
c: 1238982130.0
d: 1238982145.0
heart - DEBUG trace: Trace (IU, KMBO, filt, Z)
timerange: 2009-04-06 01:31:10 - 2009-04-06 02:18:04
delta t: 2
heart - DEBUG chopped trace: Trace (IU, KMBO, filt, Z)
timerange: 2009-04-06 01:41:20 - 2009-04-06 01:42:08
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1335
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982045.0
b: 1238982050.0
c: 1238982100.0
d: 1238982115.0
heart - DEBUG trace: Trace (II, KURK, filt, Z)
timerange: 2009-04-06 01:30:38 - 2009-04-06 02:15:06
delta t: 2
heart - DEBUG chopped trace: Trace (II, KURK, filt, Z)
timerange: 2009-04-06 01:40:50 - 2009-04-06 01:41:38
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1195
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981977.0
b: 1238981982.0
c: 1238982032.0
d: 1238982047.0
heart - DEBUG trace: Trace (ZF, ADYE, filt, Z)
timerange: 2009-04-06 01:29:32 - 2009-04-06 02:09:20
delta t: 2
heart - DEBUG chopped trace: Trace (ZF, ADYE, filt, Z)
timerange: 2009-04-06 01:39:42 - 2009-04-06 01:40:30
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1978
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982281.0
b: 1238982286.0
c: 1238982336.0
d: 1238982351.0
heart - DEBUG trace: Trace (IU, SDV, filt, Z)
timerange: 2009-04-06 01:34:36 - 2009-04-06 02:40:30
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SDV, filt, Z)
timerange: 2009-04-06 01:44:46 - 2009-04-06 01:45:34
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1120
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981941.0
b: 1238981946.0
c: 1238981996.0
d: 1238982011.0
heart - DEBUG trace: Trace (II, ARU, filt, Z)
timerange: 2009-04-06 01:28:54 - 2009-04-06 02:06:12
delta t: 2
heart - DEBUG chopped trace: Trace (II, ARU, filt, Z)
timerange: 2009-04-06 01:39:06 - 2009-04-06 01:39:54
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2105
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982315.0
b: 1238982320.0
c: 1238982370.0
d: 1238982385.0
heart - DEBUG trace: Trace (IU, SAML, filt, Z)
timerange: 2009-04-06 01:35:10 - 2009-04-06 02:45:18
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SAML, filt, Z)
timerange: 2009-04-06 01:45:20 - 2009-04-06 01:46:08
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1370
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982061.0
b: 1238982066.0
c: 1238982116.0
d: 1238982131.0
heart - DEBUG trace: Trace (II, ALE, filt, Z)
timerange: 2009-04-06 01:30:54 - 2009-04-06 02:16:32
delta t: 2
heart - DEBUG chopped trace: Trace (II, ALE, filt, Z)
timerange: 2009-04-06 01:41:06 - 2009-04-06 01:41:54
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1159
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981959.0
b: 1238981964.0
c: 1238982014.0
d: 1238982029.0
heart - DEBUG trace: Trace (II, ABKT, filt, Z)
timerange: 2009-04-06 01:29:14 - 2009-04-06 02:07:50
delta t: 2
heart - DEBUG chopped trace: Trace (II, ABKT, filt, Z)
timerange: 2009-04-06 01:39:24 - 2009-04-06 01:40:12
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1110
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981935.0
b: 1238981940.0
c: 1238981990.0
d: 1238982005.0
heart - DEBUG trace: Trace (KZ, AKTO, filt, Z)
timerange: 2009-04-06 01:28:50 - 2009-04-06 02:05:48
delta t: 2
heart - DEBUG chopped trace: Trace (KZ, AKTO, filt, Z)
timerange: 2009-04-06 01:39:00 - 2009-04-06 01:39:48
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1838
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982239.0
b: 1238982244.0
c: 1238982294.0
d: 1238982309.0
heart - DEBUG trace: Trace (IC, BJT, filt, Z)
timerange: 2009-04-06 01:33:54 - 2009-04-06 02:35:08
delta t: 2
heart - DEBUG chopped trace: Trace (IC, BJT, filt, Z)
timerange: 2009-04-06 01:44:04 - 2009-04-06 01:44:52
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2012
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982291.0
b: 1238982296.0
c: 1238982346.0
d: 1238982361.0
heart - DEBUG trace: Trace (IC, SSE, filt, Z)
timerange: 2009-04-06 01:34:44 - 2009-04-06 02:41:46
delta t: 2
heart - DEBUG chopped trace: Trace (IC, SSE, filt, Z)
timerange: 2009-04-06 01:44:56 - 2009-04-06 01:45:44
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1879
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982251.0
b: 1238982256.0
c: 1238982306.0
d: 1238982321.0
heart - DEBUG trace: Trace (IU, CHTO, filt, Z)
timerange: 2009-04-06 01:34:06 - 2009-04-06 02:36:42
delta t: 2
heart - DEBUG chopped trace: Trace (IU, CHTO, filt, Z)
timerange: 2009-04-06 01:44:16 - 2009-04-06 01:45:04
delta t: 2
heart - DEBUG Returning chopped traces ...
seismic - INFO Initializing synthesizer for "any_P_0"
heart - DEBUG Synthetics generation time: 0.017754
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982169.0
b: 1238982174.0
c: 1238982224.0
d: 1238982239.0
heart - DEBUG trace: Trace (IU, TSUM, 0, Z)
timerange: 2009-04-06 01:42:38 - 2009-04-06 01:44:18
delta t: 2
heart - DEBUG chopped trace: Trace (IU, TSUM, 0, Z)
timerange: 2009-04-06 01:42:54 - 2009-04-06 01:43:42
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982197.0
b: 1238982202.0
c: 1238982252.0
d: 1238982267.0
heart - DEBUG trace: Trace (IU, RCBR, 0, Z)
timerange: 2009-04-06 01:43:06 - 2009-04-06 01:44:46
delta t: 2
heart - DEBUG chopped trace: Trace (IU, RCBR, 0, Z)
timerange: 2009-04-06 01:43:22 - 2009-04-06 01:44:10
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982119.0
b: 1238982124.0
c: 1238982174.0
d: 1238982189.0
heart - DEBUG trace: Trace (CN, RES, 0, Z)
timerange: 2009-04-06 01:41:48 - 2009-04-06 01:43:28
delta t: 2
heart - DEBUG chopped trace: Trace (CN, RES, 0, Z)
timerange: 2009-04-06 01:42:04 - 2009-04-06 01:42:52
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982245.0
b: 1238982250.0
c: 1238982300.0
d: 1238982315.0
heart - DEBUG trace: Trace (IC, KMI, 0, Z)
timerange: 2009-04-06 01:43:54 - 2009-04-06 01:45:34
delta t: 2
heart - DEBUG chopped trace: Trace (IC, KMI, 0, Z)
timerange: 2009-04-06 01:44:10 - 2009-04-06 01:44:58
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982193.0
b: 1238982198.0
c: 1238982248.0
d: 1238982263.0
heart - DEBUG trace: Trace (IU, SSPA, 0, Z)
timerange: 2009-04-06 01:43:02 - 2009-04-06 01:44:42
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SSPA, 0, Z)
timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:06
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982331.0
b: 1238982336.0
c: 1238982386.0
d: 1238982401.0
heart - DEBUG trace: Trace (MS, BTDF, 0, Z)
timerange: 2009-04-06 01:45:20 - 2009-04-06 01:47:00
delta t: 2
heart - DEBUG chopped trace: Trace (MS, BTDF, 0, Z)
timerange: 2009-04-06 01:45:36 - 2009-04-06 01:46:24
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982209.0
b: 1238982214.0
c: 1238982264.0
d: 1238982279.0
heart - DEBUG trace: Trace (CU, ANWB, 0, Z)
timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:58
delta t: 2
heart - DEBUG chopped trace: Trace (CU, ANWB, 0, Z)
timerange: 2009-04-06 01:43:34 - 2009-04-06 01:44:22
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981993.0
b: 1238981998.0
c: 1238982048.0
d: 1238982063.0
heart - DEBUG trace: Trace (G, MBO, 0, Z)
timerange: 2009-04-06 01:39:42 - 2009-04-06 01:41:22
delta t: 2
heart - DEBUG chopped trace: Trace (G, MBO, 0, Z)
timerange: 2009-04-06 01:39:58 - 2009-04-06 01:40:46
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982309.0
b: 1238982314.0
c: 1238982364.0
d: 1238982379.0
heart - DEBUG trace: Trace (IU, TEIG, 0, Z)
timerange: 2009-04-06 01:44:58 - 2009-04-06 01:46:38
delta t: 2
heart - DEBUG chopped trace: Trace (IU, TEIG, 0, Z)
timerange: 2009-04-06 01:45:14 - 2009-04-06 01:46:02
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982039.0
b: 1238982044.0
c: 1238982094.0
d: 1238982109.0
heart - DEBUG trace: Trace (IU, KBL, 0, Z)
timerange: 2009-04-06 01:40:28 - 2009-04-06 01:42:08
delta t: 2
heart - DEBUG chopped trace: Trace (IU, KBL, 0, Z)
timerange: 2009-04-06 01:40:44 - 2009-04-06 01:41:32
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982163.0
b: 1238982168.0
c: 1238982218.0
d: 1238982233.0
heart - DEBUG trace: Trace (II, MSEY, 0, Z)
timerange: 2009-04-06 01:42:32 - 2009-04-06 01:44:12
delta t: 2
heart - DEBUG chopped trace: Trace (II, MSEY, 0, Z)
timerange: 2009-04-06 01:42:48 - 2009-04-06 01:43:36
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982001.0
b: 1238982006.0
c: 1238982056.0
d: 1238982071.0
heart - DEBUG trace: Trace (GT, DBIC, 0, Z)
timerange: 2009-04-06 01:39:50 - 2009-04-06 01:41:30
delta t: 2
heart - DEBUG chopped trace: Trace (GT, DBIC, 0, Z)
timerange: 2009-04-06 01:40:06 - 2009-04-06 01:40:54
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982215.0
b: 1238982220.0
c: 1238982270.0
d: 1238982285.0
heart - DEBUG trace: Trace (IC, HIA, 0, Z)
timerange: 2009-04-06 01:43:24 - 2009-04-06 01:45:04
delta t: 2
heart - DEBUG chopped trace: Trace (IC, HIA, 0, Z)
timerange: 2009-04-06 01:43:40 - 2009-04-06 01:44:28
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982187.0
b: 1238982192.0
c: 1238982242.0
d: 1238982257.0
heart - DEBUG trace: Trace (IU, YAK, 0, Z)
timerange: 2009-04-06 01:42:56 - 2009-04-06 01:44:36
delta t: 2
heart - DEBUG chopped trace: Trace (IU, YAK, 0, Z)
timerange: 2009-04-06 01:43:12 - 2009-04-06 01:44:00
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982295.0
b: 1238982300.0
c: 1238982350.0
d: 1238982365.0
heart - DEBUG trace: Trace (IC, QIZ, 0, Z)
timerange: 2009-04-06 01:44:44 - 2009-04-06 01:46:24
delta t: 2
heart - DEBUG chopped trace: Trace (IC, QIZ, 0, Z)
timerange: 2009-04-06 01:45:00 - 2009-04-06 01:45:48
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981981.0
b: 1238981986.0
c: 1238982036.0
d: 1238982051.0
heart - DEBUG trace: Trace (GE, KBS, 0, Z)
timerange: 2009-04-06 01:39:30 - 2009-04-06 01:41:10
delta t: 2
heart - DEBUG chopped trace: Trace (GE, KBS, 0, Z)
timerange: 2009-04-06 01:39:46 - 2009-04-06 01:40:34
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982153.0
b: 1238982158.0
c: 1238982208.0
d: 1238982223.0
heart - DEBUG trace: Trace (IU, LSZ, 0, Z)
timerange: 2009-04-06 01:42:22 - 2009-04-06 01:44:02
delta t: 2
heart - DEBUG chopped trace: Trace (IU, LSZ, 0, Z)
timerange: 2009-04-06 01:42:38 - 2009-04-06 01:43:26
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982149.0
b: 1238982154.0
c: 1238982204.0
d: 1238982219.0
heart - DEBUG trace: Trace (II, TLY, 0, Z)
timerange: 2009-04-06 01:42:18 - 2009-04-06 01:43:58
delta t: 2
heart - DEBUG chopped trace: Trace (II, TLY, 0, Z)
timerange: 2009-04-06 01:42:34 - 2009-04-06 01:43:22
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982283.0
b: 1238982288.0
c: 1238982338.0
d: 1238982353.0
heart - DEBUG trace: Trace (IU, INCN, 0, Z)
timerange: 2009-04-06 01:44:32 - 2009-04-06 01:46:12
delta t: 2
heart - DEBUG chopped trace: Trace (IU, INCN, 0, Z)
timerange: 2009-04-06 01:44:48 - 2009-04-06 01:45:36
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982255.0
b: 1238982260.0
c: 1238982310.0
d: 1238982325.0
heart - DEBUG trace: Trace (IU, DWPF, 0, Z)
timerange: 2009-04-06 01:44:04 - 2009-04-06 01:45:44
delta t: 2
heart - DEBUG chopped trace: Trace (IU, DWPF, 0, Z)
timerange: 2009-04-06 01:44:20 - 2009-04-06 01:45:08
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982107.0
b: 1238982112.0
c: 1238982162.0
d: 1238982177.0
heart - DEBUG trace: Trace (IC, WMQ, 0, Z)
timerange: 2009-04-06 01:41:36 - 2009-04-06 01:43:16
delta t: 2
heart - DEBUG chopped trace: Trace (IC, WMQ, 0, Z)
timerange: 2009-04-06 01:41:52 - 2009-04-06 01:42:40
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982043.0
b: 1238982048.0
c: 1238982098.0
d: 1238982113.0
heart - DEBUG trace: Trace (II, AAK, 0, Z)
timerange: 2009-04-06 01:40:32 - 2009-04-06 01:42:12
delta t: 2
heart - DEBUG chopped trace: Trace (II, AAK, 0, Z)
timerange: 2009-04-06 01:40:48 - 2009-04-06 01:41:36
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982235.0
b: 1238982240.0
c: 1238982290.0
d: 1238982305.0
heart - DEBUG trace: Trace (IC, XAN, 0, Z)
timerange: 2009-04-06 01:43:44 - 2009-04-06 01:45:24
delta t: 2
heart - DEBUG chopped trace: Trace (IC, XAN, 0, Z)
timerange: 2009-04-06 01:44:00 - 2009-04-06 01:44:48
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982075.0
b: 1238982080.0
c: 1238982130.0
d: 1238982145.0
heart - DEBUG trace: Trace (IU, KMBO, 0, Z)
timerange: 2009-04-06 01:41:04 - 2009-04-06 01:42:44
delta t: 2
heart - DEBUG chopped trace: Trace (IU, KMBO, 0, Z)
timerange: 2009-04-06 01:41:20 - 2009-04-06 01:42:08
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982045.0
b: 1238982050.0
c: 1238982100.0
d: 1238982115.0
heart - DEBUG trace: Trace (II, KURK, 0, Z)
timerange: 2009-04-06 01:40:34 - 2009-04-06 01:42:14
delta t: 2
heart - DEBUG chopped trace: Trace (II, KURK, 0, Z)
timerange: 2009-04-06 01:40:50 - 2009-04-06 01:41:38
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981977.0
b: 1238981982.0
c: 1238982032.0
d: 1238982047.0
heart - DEBUG trace: Trace (ZF, ADYE, 0, Z)
timerange: 2009-04-06 01:39:26 - 2009-04-06 01:41:06
delta t: 2
heart - DEBUG chopped trace: Trace (ZF, ADYE, 0, Z)
timerange: 2009-04-06 01:39:42 - 2009-04-06 01:40:30
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982281.0
b: 1238982286.0
c: 1238982336.0
d: 1238982351.0
heart - DEBUG trace: Trace (IU, SDV, 0, Z)
timerange: 2009-04-06 01:44:30 - 2009-04-06 01:46:10
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SDV, 0, Z)
timerange: 2009-04-06 01:44:46 - 2009-04-06 01:45:34
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981941.0
b: 1238981946.0
c: 1238981996.0
d: 1238982011.0
heart - DEBUG trace: Trace (II, ARU, 0, Z)
timerange: 2009-04-06 01:38:50 - 2009-04-06 01:40:30
delta t: 2
heart - DEBUG chopped trace: Trace (II, ARU, 0, Z)
timerange: 2009-04-06 01:39:06 - 2009-04-06 01:39:54
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982315.0
b: 1238982320.0
c: 1238982370.0
d: 1238982385.0
heart - DEBUG trace: Trace (IU, SAML, 0, Z)
timerange: 2009-04-06 01:45:04 - 2009-04-06 01:46:44
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SAML, 0, Z)
timerange: 2009-04-06 01:45:20 - 2009-04-06 01:46:08
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982061.0
b: 1238982066.0
c: 1238982116.0
d: 1238982131.0
heart - DEBUG trace: Trace (II, ALE, 0, Z)
timerange: 2009-04-06 01:40:50 - 2009-04-06 01:42:30
delta t: 2
heart - DEBUG chopped trace: Trace (II, ALE, 0, Z)
timerange: 2009-04-06 01:41:06 - 2009-04-06 01:41:54
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981959.0
b: 1238981964.0
c: 1238982014.0
d: 1238982029.0
heart - DEBUG trace: Trace (II, ABKT, 0, Z)
timerange: 2009-04-06 01:39:08 - 2009-04-06 01:40:48
delta t: 2
heart - DEBUG chopped trace: Trace (II, ABKT, 0, Z)
timerange: 2009-04-06 01:39:24 - 2009-04-06 01:40:12
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981935.0
b: 1238981940.0
c: 1238981990.0
d: 1238982005.0
heart - DEBUG trace: Trace (KZ, AKTO, 0, Z)
timerange: 2009-04-06 01:38:44 - 2009-04-06 01:40:24
delta t: 2
heart - DEBUG chopped trace: Trace (KZ, AKTO, 0, Z)
timerange: 2009-04-06 01:39:00 - 2009-04-06 01:39:48
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982239.0
b: 1238982244.0
c: 1238982294.0
d: 1238982309.0
heart - DEBUG trace: Trace (IC, BJT, 0, Z)
timerange: 2009-04-06 01:43:48 - 2009-04-06 01:45:28
delta t: 2
heart - DEBUG chopped trace: Trace (IC, BJT, 0, Z)
timerange: 2009-04-06 01:44:04 - 2009-04-06 01:44:52
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982291.0
b: 1238982296.0
c: 1238982346.0
d: 1238982361.0
heart - DEBUG trace: Trace (IC, SSE, 0, Z)
timerange: 2009-04-06 01:44:40 - 2009-04-06 01:46:20
delta t: 2
heart - DEBUG chopped trace: Trace (IC, SSE, 0, Z)
timerange: 2009-04-06 01:44:56 - 2009-04-06 01:45:44
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982251.0
b: 1238982256.0
c: 1238982306.0
d: 1238982321.0
heart - DEBUG trace: Trace (IU, CHTO, 0, Z)
timerange: 2009-04-06 01:44:00 - 2009-04-06 01:45:40
delta t: 2
heart - DEBUG chopped trace: Trace (IU, CHTO, 0, Z)
timerange: 2009-04-06 01:44:16 - 2009-04-06 01:45:04
delta t: 2
heart - DEBUG Post-process time 0.023978
heart - DEBUG Stack traces time 0.000000
heart - DEBUG Returning...
seismic - DEBUG Teleseismic forward model on test model takes: 0.827361
models - INFO Model building was successful!
models - INFO Using "bin" backend to store samples!
models - INFO ... Initiate Metropolis for Parallel Tempering...
proposal_distribution: Normal, tune_interval=10, n_chains=8
models.base - INFO ... Starting Parallel Tempering ...
distributed - INFO MPI is installed!
distributed - INFO Detected python3
distributed - INFO Done initialising mpi runner
distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py
distributed - DEBUG Running mpi with arguments: -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
Traceback (most recent call last):
File "/home/lab/anaconda3/envs/beat/bin/beat", line 11, in
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 1686, in main
globals()'command_' + command
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 775, in command_sample
sample(step, problem)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/base.py", line 196, in sample
record_worker_chains=pa.record_worker_chains)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/pt.py", line 836, in pt_sample
project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 226, in run_mpi_sampler
samplerdir, n_jobs=n_jobs, loglevel=loglevel, project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 158, in run
in the directory %s'''.lstrip() % (program, self.tempdir))
beat.sampler.distributed.MPIError: mpiexec has been invoked as "mpiexec"
in the directory /tmp/mpiexec-7bw3hmzo
distributed - WARNING not removing temporary directory: /tmp/mpiexec-7bw3hmzo
------------------------------------------------------------*-
3) Running:
mpiexec -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
in the temporary directory: /tmp/mpiexec-7bw3hmzo
I got:
(beat) lab@lab:/tmp/mpiexec-7bw3hmzo$ mpiexec -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
There are not enough slots available in the system to satisfy the 9
slots that were requested by the application:
python3
Either request fewer slots for your application, or make more slots
available for use.
A "slot" is the Open MPI term for an allocatable unit where we can
launch a process. The number of slots available are defined by the
environment in which Open MPI processes are run:
processor cores if not provided)
hostname (N defaults to 1 if not provided)
RM is present, Open MPI defaults to the number of processor cores
In all the above cases, if you want Open MPI to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.
Alternatively, you can use the --oversubscribe option to ignore the
number of available slots when deciding the number of processes to
launch.
------------------------------------------------------------*-
4) running the line:
mpiexec -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc --use-hwthread-cpus
Got:
(beat) lab@lab:/tmp/mpiexec-7bw3hmzo$ mpiexec -n 9 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py --use-hwthread-cpus
There are not enough slots available in the system to satisfy the 9
slots that were requested by the application:
python3
Either request fewer slots for your application, or make more slots
available for use.
A "slot" is the Open MPI term for an allocatable unit where we can
launch a process. The number of slots available are defined by the
environment in which Open MPI processes are run:
processor cores if not provided)
hostname (N defaults to 1 if not provided)
RM is present, Open MPI defaults to the number of processor cores
In all the above cases, if you want Open MPI to default to the number
of hardware threads instead of the number of processor cores, use the
--use-hwthread-cpus option.
Alternatively, you can use the --oversubscribe option to ignore the
number of available slots when deciding the number of processes to
launch.
------------------------------------------------------------*-
5) Now, Running the second example with n_chains=5, I got:
(beat) lab@lab:~$ beat sample Laquila_dc --loglevel=debug
config - INFO All hierarchicals ok!
config - INFO All hyperparameters ok!
config - INFO All priors ok!
models - INFO ... Initialising Geometry Optimizer ...
models - INFO Analysing problem ...
models - INFO ---------------------
seismic - DEBUG Setting up seismic structure ...
utility - DEBUG Valid distance range: [30.000000, 90.000000]!
utility - DEBUG Distance of station TSUM: 61.349855 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station RCBR: 65.567308 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station RES: 54.143590 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KMI: 73.109289 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station SSPA: 64.946397 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station BTDF: 89.437223 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station ANWB: 67.439618 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station MBO: 38.167185 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station TEIG: 84.816714 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KBL: 43.633450 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station MSEY: 60.417579 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station DBIC: 38.986606 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station HIA: 68.244425 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station YAK: 64.054778 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station QIZ: 82.062195 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KBS: 36.698309 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station LSZ: 58.931762 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station TLY: 58.456908 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station INCN: 79.971426 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station DWPF: 74.925103 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station WMQ: 52.564826 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station AAK: 44.192670 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station XAN: 71.681441 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KMBO: 48.335702 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station KURK: 44.334011 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station ADYE: 36.168619 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station SDV: 79.664513 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station ARU: 31.995457 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station SAML: 86.199701 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station ALE: 46.405043 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station OTAV: 91.209244 [deg]
utility - DEBUG Outside defined distance range!
utility - DEBUG Distance of station ABKT: 34.203018 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station AKTO: 31.451507 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station BJT: 72.303754 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station SSE: 81.349799 [deg]
utility - DEBUG Inside defined distance range!
utility - DEBUG Distance of station CHTO: 74.397519 [deg]
utility - DEBUG Inside defined distance range!
heart - INFO Consistent number of datasets and targets in any_P_0 wavemap!
heart - INFO Number of seismic datasets for wavemap: any_P_0: 35
models - INFO ... Building model ...
models - DEBUG Optimization for 1 sources
models - INFO Optimization for 35 hyperparameters in total!
seismic - INFO Seismic optimization on:
depth, dip, duration, east_shift, magnitude, north_shift, rake, strike, time
seismic - INFO Retrieving seismic data-covariances with structure "variance" for any_P_0 ...
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 61.349855 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 65.567308 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 54.143590 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 73.109289 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 64.946397 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 89.437223 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 67.439618 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 38.167185 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 84.816714 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 43.633450 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 60.417579 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 38.986606 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 68.244425 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 64.054778 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 82.062195 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 36.698309 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 58.931762 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 58.456908 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 79.971426 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 74.925103 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 52.564826 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 44.192670 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 71.681441 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 48.335702 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 44.334011 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 36.168619 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 79.664513 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 31.995457 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 86.199701 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 46.405043 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 34.203018 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 31.451507 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 72.303754 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 81.349799 [deg]
heart - DEBUG Wavename not specified using first tabulated phase! any_P
heart - DEBUG Arrival time for wavename "any_P" distance 74.397519 [deg]
seismic - INFO Preparing data of "any_P_0" for optimization
heart - DEBUG Arrival time for wavename "any_P" distance 61.349855 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 65.567308 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 54.143590 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 73.109289 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 64.946397 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 89.437223 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 67.439618 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 38.167185 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 84.816714 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 43.633450 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 60.417579 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 38.986606 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 68.244425 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 64.054778 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 82.062195 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 36.698309 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 58.931762 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 58.456908 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 79.971426 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 74.925103 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 52.564826 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 44.192670 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 71.681441 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 48.335702 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 44.334011 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 36.168619 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 79.664513 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 31.995457 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 86.199701 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 46.405043 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 34.203018 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 31.451507 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 72.303754 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 81.349799 [deg]
heart - DEBUG Arrival time for wavename "any_P" distance 74.397519 [deg]
heart - DEBUG Pre-processing data ...
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1641
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982169.0
b: 1238982174.0
c: 1238982224.0
d: 1238982239.0
heart - DEBUG trace: Trace (IU, TSUM, filt, Z)
timerange: 2009-04-06 01:32:44 - 2009-04-06 02:27:24
delta t: 2
heart - DEBUG chopped trace: Trace (IU, TSUM, filt, Z)
timerange: 2009-04-06 01:42:54 - 2009-04-06 01:43:42
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1716
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982197.0
b: 1238982202.0
c: 1238982252.0
d: 1238982267.0
heart - DEBUG trace: Trace (IU, RCBR, filt, Z)
timerange: 2009-04-06 01:33:12 - 2009-04-06 02:30:22
delta t: 2
heart - DEBUG chopped trace: Trace (IU, RCBR, filt, Z)
timerange: 2009-04-06 01:43:22 - 2009-04-06 01:44:10
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1508
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982119.0
b: 1238982124.0
c: 1238982174.0
d: 1238982189.0
heart - DEBUG trace: Trace (CN, RES, filt, Z)
timerange: 2009-04-06 01:31:52 - 2009-04-06 02:22:06
delta t: 2
heart - DEBUG chopped trace: Trace (CN, RES, filt, Z)
timerange: 2009-04-06 01:42:04 - 2009-04-06 01:42:52
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1853
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982245.0
b: 1238982250.0
c: 1238982300.0
d: 1238982315.0
heart - DEBUG trace: Trace (IC, KMI, filt, Z)
timerange: 2009-04-06 01:34:00 - 2009-04-06 02:35:44
delta t: 2
heart - DEBUG chopped trace: Trace (IC, KMI, filt, Z)
timerange: 2009-04-06 01:44:10 - 2009-04-06 01:44:58
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1702
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982193.0
b: 1238982198.0
c: 1238982248.0
d: 1238982263.0
heart - DEBUG trace: Trace (IU, SSPA, filt, Z)
timerange: 2009-04-06 01:33:08 - 2009-04-06 02:29:50
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SSPA, filt, Z)
timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:06
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2169
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982331.0
b: 1238982336.0
c: 1238982386.0
d: 1238982401.0
heart - DEBUG trace: Trace (MS, BTDF, filt, Z)
timerange: 2009-04-06 01:35:26 - 2009-04-06 02:47:42
delta t: 2
heart - DEBUG chopped trace: Trace (MS, BTDF, filt, Z)
timerange: 2009-04-06 01:45:36 - 2009-04-06 01:46:24
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1748
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982209.0
b: 1238982214.0
c: 1238982264.0
d: 1238982279.0
heart - DEBUG trace: Trace (CU, ANWB, filt, Z)
timerange: 2009-04-06 01:33:24 - 2009-04-06 02:31:38
delta t: 2
heart - DEBUG chopped trace: Trace (CU, ANWB, filt, Z)
timerange: 2009-04-06 01:43:34 - 2009-04-06 01:44:22
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1229
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981993.0
b: 1238981998.0
c: 1238982048.0
d: 1238982063.0
heart - DEBUG trace: Trace (G, MBO, filt, Z)
timerange: 2009-04-06 01:29:48 - 2009-04-06 02:10:44
delta t: 2
heart - DEBUG chopped trace: Trace (G, MBO, filt, Z)
timerange: 2009-04-06 01:39:58 - 2009-04-06 01:40:46
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2078
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982309.0
b: 1238982314.0
c: 1238982364.0
d: 1238982379.0
heart - DEBUG trace: Trace (IU, TEIG, filt, Z)
timerange: 2009-04-06 01:35:02 - 2009-04-06 02:44:16
delta t: 2
heart - DEBUG chopped trace: Trace (IU, TEIG, filt, Z)
timerange: 2009-04-06 01:45:14 - 2009-04-06 01:46:02
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1324
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982039.0
b: 1238982044.0
c: 1238982094.0
d: 1238982109.0
heart - DEBUG trace: Trace (IU, KBL, filt, Z)
timerange: 2009-04-06 01:30:32 - 2009-04-06 02:14:38
delta t: 2
heart - DEBUG chopped trace: Trace (IU, KBL, filt, Z)
timerange: 2009-04-06 01:40:44 - 2009-04-06 01:41:32
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1624
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982163.0
b: 1238982168.0
c: 1238982218.0
d: 1238982233.0
heart - DEBUG trace: Trace (II, MSEY, filt, Z)
timerange: 2009-04-06 01:32:38 - 2009-04-06 02:26:44
delta t: 2
heart - DEBUG chopped trace: Trace (II, MSEY, filt, Z)
timerange: 2009-04-06 01:42:48 - 2009-04-06 01:43:36
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1244
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982001.0
b: 1238982006.0
c: 1238982056.0
d: 1238982071.0
heart - DEBUG trace: Trace (GT, DBIC, filt, Z)
timerange: 2009-04-06 01:29:54 - 2009-04-06 02:11:20
delta t: 2
heart - DEBUG chopped trace: Trace (GT, DBIC, filt, Z)
timerange: 2009-04-06 01:40:06 - 2009-04-06 01:40:54
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1763
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982215.0
b: 1238982220.0
c: 1238982270.0
d: 1238982285.0
heart - DEBUG trace: Trace (IC, HIA, filt, Z)
timerange: 2009-04-06 01:33:30 - 2009-04-06 02:32:14
delta t: 2
heart - DEBUG chopped trace: Trace (IC, HIA, filt, Z)
timerange: 2009-04-06 01:43:40 - 2009-04-06 01:44:28
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1686
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982187.0
b: 1238982192.0
c: 1238982242.0
d: 1238982257.0
heart - DEBUG trace: Trace (IU, YAK, filt, Z)
timerange: 2009-04-06 01:33:02 - 2009-04-06 02:29:12
delta t: 2
heart - DEBUG chopped trace: Trace (IU, YAK, filt, Z)
timerange: 2009-04-06 01:43:12 - 2009-04-06 01:44:00
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2026
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982295.0
b: 1238982300.0
c: 1238982350.0
d: 1238982365.0
heart - DEBUG trace: Trace (IC, QIZ, filt, Z)
timerange: 2009-04-06 01:34:48 - 2009-04-06 02:42:18
delta t: 2
heart - DEBUG chopped trace: Trace (IC, QIZ, filt, Z)
timerange: 2009-04-06 01:45:00 - 2009-04-06 01:45:48
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1201
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981981.0
b: 1238981986.0
c: 1238982036.0
d: 1238982051.0
heart - DEBUG trace: Trace (GE, KBS, filt, Z)
timerange: 2009-04-06 01:29:34 - 2009-04-06 02:09:34
delta t: 2
heart - DEBUG chopped trace: Trace (GE, KBS, filt, Z)
timerange: 2009-04-06 01:39:46 - 2009-04-06 01:40:34
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1598
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982153.0
b: 1238982158.0
c: 1238982208.0
d: 1238982223.0
heart - DEBUG trace: Trace (IU, LSZ, filt, Z)
timerange: 2009-04-06 01:32:28 - 2009-04-06 02:25:42
delta t: 2
heart - DEBUG chopped trace: Trace (IU, LSZ, filt, Z)
timerange: 2009-04-06 01:42:38 - 2009-04-06 01:43:26
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1586
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982149.0
b: 1238982154.0
c: 1238982204.0
d: 1238982219.0
heart - DEBUG trace: Trace (II, TLY, filt, Z)
timerange: 2009-04-06 01:32:24 - 2009-04-06 02:25:14
delta t: 2
heart - DEBUG chopped trace: Trace (II, TLY, filt, Z)
timerange: 2009-04-06 01:42:34 - 2009-04-06 01:43:22
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1984
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982283.0
b: 1238982288.0
c: 1238982338.0
d: 1238982353.0
heart - DEBUG trace: Trace (IU, INCN, filt, Z)
timerange: 2009-04-06 01:34:38 - 2009-04-06 02:40:44
delta t: 2
heart - DEBUG chopped trace: Trace (IU, INCN, filt, Z)
timerange: 2009-04-06 01:44:48 - 2009-04-06 01:45:36
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1888
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982255.0
b: 1238982260.0
c: 1238982310.0
d: 1238982325.0
heart - DEBUG trace: Trace (IU, DWPF, filt, Z)
timerange: 2009-04-06 01:34:08 - 2009-04-06 02:37:02
delta t: 2
heart - DEBUG chopped trace: Trace (IU, DWPF, filt, Z)
timerange: 2009-04-06 01:44:20 - 2009-04-06 01:45:08
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1481
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982107.0
b: 1238982112.0
c: 1238982162.0
d: 1238982177.0
heart - DEBUG trace: Trace (IC, WMQ, filt, Z)
timerange: 2009-04-06 01:31:42 - 2009-04-06 02:21:02
delta t: 2
heart - DEBUG chopped trace: Trace (IC, WMQ, filt, Z)
timerange: 2009-04-06 01:41:52 - 2009-04-06 01:42:40
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1334
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982043.0
b: 1238982048.0
c: 1238982098.0
d: 1238982113.0
heart - DEBUG trace: Trace (II, AAK, filt, Z)
timerange: 2009-04-06 01:30:36 - 2009-04-06 02:15:02
delta t: 2
heart - DEBUG chopped trace: Trace (II, AAK, filt, Z)
timerange: 2009-04-06 01:40:48 - 2009-04-06 01:41:36
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1827
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982235.0
b: 1238982240.0
c: 1238982290.0
d: 1238982305.0
heart - DEBUG trace: Trace (IC, XAN, filt, Z)
timerange: 2009-04-06 01:33:50 - 2009-04-06 02:34:42
delta t: 2
heart - DEBUG chopped trace: Trace (IC, XAN, filt, Z)
timerange: 2009-04-06 01:44:00 - 2009-04-06 01:44:48
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1408
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982075.0
b: 1238982080.0
c: 1238982130.0
d: 1238982145.0
heart - DEBUG trace: Trace (IU, KMBO, filt, Z)
timerange: 2009-04-06 01:31:10 - 2009-04-06 02:18:04
delta t: 2
heart - DEBUG chopped trace: Trace (IU, KMBO, filt, Z)
timerange: 2009-04-06 01:41:20 - 2009-04-06 01:42:08
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1335
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982045.0
b: 1238982050.0
c: 1238982100.0
d: 1238982115.0
heart - DEBUG trace: Trace (II, KURK, filt, Z)
timerange: 2009-04-06 01:30:38 - 2009-04-06 02:15:06
delta t: 2
heart - DEBUG chopped trace: Trace (II, KURK, filt, Z)
timerange: 2009-04-06 01:40:50 - 2009-04-06 01:41:38
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1195
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981977.0
b: 1238981982.0
c: 1238982032.0
d: 1238982047.0
heart - DEBUG trace: Trace (ZF, ADYE, filt, Z)
timerange: 2009-04-06 01:29:32 - 2009-04-06 02:09:20
delta t: 2
heart - DEBUG chopped trace: Trace (ZF, ADYE, filt, Z)
timerange: 2009-04-06 01:39:42 - 2009-04-06 01:40:30
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1978
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982281.0
b: 1238982286.0
c: 1238982336.0
d: 1238982351.0
heart - DEBUG trace: Trace (IU, SDV, filt, Z)
timerange: 2009-04-06 01:34:36 - 2009-04-06 02:40:30
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SDV, filt, Z)
timerange: 2009-04-06 01:44:46 - 2009-04-06 01:45:34
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1120
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981941.0
b: 1238981946.0
c: 1238981996.0
d: 1238982011.0
heart - DEBUG trace: Trace (II, ARU, filt, Z)
timerange: 2009-04-06 01:28:54 - 2009-04-06 02:06:12
delta t: 2
heart - DEBUG chopped trace: Trace (II, ARU, filt, Z)
timerange: 2009-04-06 01:39:06 - 2009-04-06 01:39:54
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2105
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982315.0
b: 1238982320.0
c: 1238982370.0
d: 1238982385.0
heart - DEBUG trace: Trace (IU, SAML, filt, Z)
timerange: 2009-04-06 01:35:10 - 2009-04-06 02:45:18
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SAML, filt, Z)
timerange: 2009-04-06 01:45:20 - 2009-04-06 01:46:08
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1370
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982061.0
b: 1238982066.0
c: 1238982116.0
d: 1238982131.0
heart - DEBUG trace: Trace (II, ALE, filt, Z)
timerange: 2009-04-06 01:30:54 - 2009-04-06 02:16:32
delta t: 2
heart - DEBUG chopped trace: Trace (II, ALE, filt, Z)
timerange: 2009-04-06 01:41:06 - 2009-04-06 01:41:54
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1159
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981959.0
b: 1238981964.0
c: 1238982014.0
d: 1238982029.0
heart - DEBUG trace: Trace (II, ABKT, filt, Z)
timerange: 2009-04-06 01:29:14 - 2009-04-06 02:07:50
delta t: 2
heart - DEBUG chopped trace: Trace (II, ABKT, filt, Z)
timerange: 2009-04-06 01:39:24 - 2009-04-06 01:40:12
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1110
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981935.0
b: 1238981940.0
c: 1238981990.0
d: 1238982005.0
heart - DEBUG trace: Trace (KZ, AKTO, filt, Z)
timerange: 2009-04-06 01:28:50 - 2009-04-06 02:05:48
delta t: 2
heart - DEBUG chopped trace: Trace (KZ, AKTO, filt, Z)
timerange: 2009-04-06 01:39:00 - 2009-04-06 01:39:48
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1838
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982239.0
b: 1238982244.0
c: 1238982294.0
d: 1238982309.0
heart - DEBUG trace: Trace (IC, BJT, filt, Z)
timerange: 2009-04-06 01:33:54 - 2009-04-06 02:35:08
delta t: 2
heart - DEBUG chopped trace: Trace (IC, BJT, filt, Z)
timerange: 2009-04-06 01:44:04 - 2009-04-06 01:44:52
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 2012
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982291.0
b: 1238982296.0
c: 1238982346.0
d: 1238982361.0
heart - DEBUG trace: Trace (IC, SSE, filt, Z)
timerange: 2009-04-06 01:34:44 - 2009-04-06 02:41:46
delta t: 2
heart - DEBUG chopped trace: Trace (IC, SSE, filt, Z)
timerange: 2009-04-06 01:44:56 - 2009-04-06 01:45:44
delta t: 2
heart - DEBUG Filtering, tapering, chopping ... trace_samples: 1879
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982251.0
b: 1238982256.0
c: 1238982306.0
d: 1238982321.0
heart - DEBUG trace: Trace (IU, CHTO, filt, Z)
timerange: 2009-04-06 01:34:06 - 2009-04-06 02:36:42
delta t: 2
heart - DEBUG chopped trace: Trace (IU, CHTO, filt, Z)
timerange: 2009-04-06 01:44:16 - 2009-04-06 01:45:04
delta t: 2
heart - DEBUG Returning chopped traces ...
seismic - INFO Initializing synthesizer for "any_P_0"
heart - DEBUG Synthetics generation time: 0.017995
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982169.0
b: 1238982174.0
c: 1238982224.0
d: 1238982239.0
heart - DEBUG trace: Trace (IU, TSUM, 0, Z)
timerange: 2009-04-06 01:42:38 - 2009-04-06 01:44:18
delta t: 2
heart - DEBUG chopped trace: Trace (IU, TSUM, 0, Z)
timerange: 2009-04-06 01:42:54 - 2009-04-06 01:43:42
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982197.0
b: 1238982202.0
c: 1238982252.0
d: 1238982267.0
heart - DEBUG trace: Trace (IU, RCBR, 0, Z)
timerange: 2009-04-06 01:43:06 - 2009-04-06 01:44:46
delta t: 2
heart - DEBUG chopped trace: Trace (IU, RCBR, 0, Z)
timerange: 2009-04-06 01:43:22 - 2009-04-06 01:44:10
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982119.0
b: 1238982124.0
c: 1238982174.0
d: 1238982189.0
heart - DEBUG trace: Trace (CN, RES, 0, Z)
timerange: 2009-04-06 01:41:48 - 2009-04-06 01:43:28
delta t: 2
heart - DEBUG chopped trace: Trace (CN, RES, 0, Z)
timerange: 2009-04-06 01:42:04 - 2009-04-06 01:42:52
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982245.0
b: 1238982250.0
c: 1238982300.0
d: 1238982315.0
heart - DEBUG trace: Trace (IC, KMI, 0, Z)
timerange: 2009-04-06 01:43:54 - 2009-04-06 01:45:34
delta t: 2
heart - DEBUG chopped trace: Trace (IC, KMI, 0, Z)
timerange: 2009-04-06 01:44:10 - 2009-04-06 01:44:58
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982193.0
b: 1238982198.0
c: 1238982248.0
d: 1238982263.0
heart - DEBUG trace: Trace (IU, SSPA, 0, Z)
timerange: 2009-04-06 01:43:02 - 2009-04-06 01:44:42
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SSPA, 0, Z)
timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:06
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982331.0
b: 1238982336.0
c: 1238982386.0
d: 1238982401.0
heart - DEBUG trace: Trace (MS, BTDF, 0, Z)
timerange: 2009-04-06 01:45:20 - 2009-04-06 01:47:00
delta t: 2
heart - DEBUG chopped trace: Trace (MS, BTDF, 0, Z)
timerange: 2009-04-06 01:45:36 - 2009-04-06 01:46:24
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982209.0
b: 1238982214.0
c: 1238982264.0
d: 1238982279.0
heart - DEBUG trace: Trace (CU, ANWB, 0, Z)
timerange: 2009-04-06 01:43:18 - 2009-04-06 01:44:58
delta t: 2
heart - DEBUG chopped trace: Trace (CU, ANWB, 0, Z)
timerange: 2009-04-06 01:43:34 - 2009-04-06 01:44:22
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981993.0
b: 1238981998.0
c: 1238982048.0
d: 1238982063.0
heart - DEBUG trace: Trace (G, MBO, 0, Z)
timerange: 2009-04-06 01:39:42 - 2009-04-06 01:41:22
delta t: 2
heart - DEBUG chopped trace: Trace (G, MBO, 0, Z)
timerange: 2009-04-06 01:39:58 - 2009-04-06 01:40:46
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982309.0
b: 1238982314.0
c: 1238982364.0
d: 1238982379.0
heart - DEBUG trace: Trace (IU, TEIG, 0, Z)
timerange: 2009-04-06 01:44:58 - 2009-04-06 01:46:38
delta t: 2
heart - DEBUG chopped trace: Trace (IU, TEIG, 0, Z)
timerange: 2009-04-06 01:45:14 - 2009-04-06 01:46:02
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982039.0
b: 1238982044.0
c: 1238982094.0
d: 1238982109.0
heart - DEBUG trace: Trace (IU, KBL, 0, Z)
timerange: 2009-04-06 01:40:28 - 2009-04-06 01:42:08
delta t: 2
heart - DEBUG chopped trace: Trace (IU, KBL, 0, Z)
timerange: 2009-04-06 01:40:44 - 2009-04-06 01:41:32
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982163.0
b: 1238982168.0
c: 1238982218.0
d: 1238982233.0
heart - DEBUG trace: Trace (II, MSEY, 0, Z)
timerange: 2009-04-06 01:42:32 - 2009-04-06 01:44:12
delta t: 2
heart - DEBUG chopped trace: Trace (II, MSEY, 0, Z)
timerange: 2009-04-06 01:42:48 - 2009-04-06 01:43:36
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982001.0
b: 1238982006.0
c: 1238982056.0
d: 1238982071.0
heart - DEBUG trace: Trace (GT, DBIC, 0, Z)
timerange: 2009-04-06 01:39:50 - 2009-04-06 01:41:30
delta t: 2
heart - DEBUG chopped trace: Trace (GT, DBIC, 0, Z)
timerange: 2009-04-06 01:40:06 - 2009-04-06 01:40:54
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982215.0
b: 1238982220.0
c: 1238982270.0
d: 1238982285.0
heart - DEBUG trace: Trace (IC, HIA, 0, Z)
timerange: 2009-04-06 01:43:24 - 2009-04-06 01:45:04
delta t: 2
heart - DEBUG chopped trace: Trace (IC, HIA, 0, Z)
timerange: 2009-04-06 01:43:40 - 2009-04-06 01:44:28
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982187.0
b: 1238982192.0
c: 1238982242.0
d: 1238982257.0
heart - DEBUG trace: Trace (IU, YAK, 0, Z)
timerange: 2009-04-06 01:42:56 - 2009-04-06 01:44:36
delta t: 2
heart - DEBUG chopped trace: Trace (IU, YAK, 0, Z)
timerange: 2009-04-06 01:43:12 - 2009-04-06 01:44:00
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982295.0
b: 1238982300.0
c: 1238982350.0
d: 1238982365.0
heart - DEBUG trace: Trace (IC, QIZ, 0, Z)
timerange: 2009-04-06 01:44:44 - 2009-04-06 01:46:24
delta t: 2
heart - DEBUG chopped trace: Trace (IC, QIZ, 0, Z)
timerange: 2009-04-06 01:45:00 - 2009-04-06 01:45:48
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981981.0
b: 1238981986.0
c: 1238982036.0
d: 1238982051.0
heart - DEBUG trace: Trace (GE, KBS, 0, Z)
timerange: 2009-04-06 01:39:30 - 2009-04-06 01:41:10
delta t: 2
heart - DEBUG chopped trace: Trace (GE, KBS, 0, Z)
timerange: 2009-04-06 01:39:46 - 2009-04-06 01:40:34
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982153.0
b: 1238982158.0
c: 1238982208.0
d: 1238982223.0
heart - DEBUG trace: Trace (IU, LSZ, 0, Z)
timerange: 2009-04-06 01:42:22 - 2009-04-06 01:44:02
delta t: 2
heart - DEBUG chopped trace: Trace (IU, LSZ, 0, Z)
timerange: 2009-04-06 01:42:38 - 2009-04-06 01:43:26
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982149.0
b: 1238982154.0
c: 1238982204.0
d: 1238982219.0
heart - DEBUG trace: Trace (II, TLY, 0, Z)
timerange: 2009-04-06 01:42:18 - 2009-04-06 01:43:58
delta t: 2
heart - DEBUG chopped trace: Trace (II, TLY, 0, Z)
timerange: 2009-04-06 01:42:34 - 2009-04-06 01:43:22
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982283.0
b: 1238982288.0
c: 1238982338.0
d: 1238982353.0
heart - DEBUG trace: Trace (IU, INCN, 0, Z)
timerange: 2009-04-06 01:44:32 - 2009-04-06 01:46:12
delta t: 2
heart - DEBUG chopped trace: Trace (IU, INCN, 0, Z)
timerange: 2009-04-06 01:44:48 - 2009-04-06 01:45:36
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982255.0
b: 1238982260.0
c: 1238982310.0
d: 1238982325.0
heart - DEBUG trace: Trace (IU, DWPF, 0, Z)
timerange: 2009-04-06 01:44:04 - 2009-04-06 01:45:44
delta t: 2
heart - DEBUG chopped trace: Trace (IU, DWPF, 0, Z)
timerange: 2009-04-06 01:44:20 - 2009-04-06 01:45:08
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982107.0
b: 1238982112.0
c: 1238982162.0
d: 1238982177.0
heart - DEBUG trace: Trace (IC, WMQ, 0, Z)
timerange: 2009-04-06 01:41:36 - 2009-04-06 01:43:16
delta t: 2
heart - DEBUG chopped trace: Trace (IC, WMQ, 0, Z)
timerange: 2009-04-06 01:41:52 - 2009-04-06 01:42:40
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982043.0
b: 1238982048.0
c: 1238982098.0
d: 1238982113.0
heart - DEBUG trace: Trace (II, AAK, 0, Z)
timerange: 2009-04-06 01:40:32 - 2009-04-06 01:42:12
delta t: 2
heart - DEBUG chopped trace: Trace (II, AAK, 0, Z)
timerange: 2009-04-06 01:40:48 - 2009-04-06 01:41:36
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982235.0
b: 1238982240.0
c: 1238982290.0
d: 1238982305.0
heart - DEBUG trace: Trace (IC, XAN, 0, Z)
timerange: 2009-04-06 01:43:44 - 2009-04-06 01:45:24
delta t: 2
heart - DEBUG chopped trace: Trace (IC, XAN, 0, Z)
timerange: 2009-04-06 01:44:00 - 2009-04-06 01:44:48
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982075.0
b: 1238982080.0
c: 1238982130.0
d: 1238982145.0
heart - DEBUG trace: Trace (IU, KMBO, 0, Z)
timerange: 2009-04-06 01:41:04 - 2009-04-06 01:42:44
delta t: 2
heart - DEBUG chopped trace: Trace (IU, KMBO, 0, Z)
timerange: 2009-04-06 01:41:20 - 2009-04-06 01:42:08
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982045.0
b: 1238982050.0
c: 1238982100.0
d: 1238982115.0
heart - DEBUG trace: Trace (II, KURK, 0, Z)
timerange: 2009-04-06 01:40:34 - 2009-04-06 01:42:14
delta t: 2
heart - DEBUG chopped trace: Trace (II, KURK, 0, Z)
timerange: 2009-04-06 01:40:50 - 2009-04-06 01:41:38
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981977.0
b: 1238981982.0
c: 1238982032.0
d: 1238982047.0
heart - DEBUG trace: Trace (ZF, ADYE, 0, Z)
timerange: 2009-04-06 01:39:26 - 2009-04-06 01:41:06
delta t: 2
heart - DEBUG chopped trace: Trace (ZF, ADYE, 0, Z)
timerange: 2009-04-06 01:39:42 - 2009-04-06 01:40:30
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982281.0
b: 1238982286.0
c: 1238982336.0
d: 1238982351.0
heart - DEBUG trace: Trace (IU, SDV, 0, Z)
timerange: 2009-04-06 01:44:30 - 2009-04-06 01:46:10
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SDV, 0, Z)
timerange: 2009-04-06 01:44:46 - 2009-04-06 01:45:34
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981941.0
b: 1238981946.0
c: 1238981996.0
d: 1238982011.0
heart - DEBUG trace: Trace (II, ARU, 0, Z)
timerange: 2009-04-06 01:38:50 - 2009-04-06 01:40:30
delta t: 2
heart - DEBUG chopped trace: Trace (II, ARU, 0, Z)
timerange: 2009-04-06 01:39:06 - 2009-04-06 01:39:54
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982315.0
b: 1238982320.0
c: 1238982370.0
d: 1238982385.0
heart - DEBUG trace: Trace (IU, SAML, 0, Z)
timerange: 2009-04-06 01:45:04 - 2009-04-06 01:46:44
delta t: 2
heart - DEBUG chopped trace: Trace (IU, SAML, 0, Z)
timerange: 2009-04-06 01:45:20 - 2009-04-06 01:46:08
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982061.0
b: 1238982066.0
c: 1238982116.0
d: 1238982131.0
heart - DEBUG trace: Trace (II, ALE, 0, Z)
timerange: 2009-04-06 01:40:50 - 2009-04-06 01:42:30
delta t: 2
heart - DEBUG chopped trace: Trace (II, ALE, 0, Z)
timerange: 2009-04-06 01:41:06 - 2009-04-06 01:41:54
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981959.0
b: 1238981964.0
c: 1238982014.0
d: 1238982029.0
heart - DEBUG trace: Trace (II, ABKT, 0, Z)
timerange: 2009-04-06 01:39:08 - 2009-04-06 01:40:48
delta t: 2
heart - DEBUG chopped trace: Trace (II, ABKT, 0, Z)
timerange: 2009-04-06 01:39:24 - 2009-04-06 01:40:12
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238981935.0
b: 1238981940.0
c: 1238981990.0
d: 1238982005.0
heart - DEBUG trace: Trace (KZ, AKTO, 0, Z)
timerange: 2009-04-06 01:38:44 - 2009-04-06 01:40:24
delta t: 2
heart - DEBUG chopped trace: Trace (KZ, AKTO, 0, Z)
timerange: 2009-04-06 01:39:00 - 2009-04-06 01:39:48
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982239.0
b: 1238982244.0
c: 1238982294.0
d: 1238982309.0
heart - DEBUG trace: Trace (IC, BJT, 0, Z)
timerange: 2009-04-06 01:43:48 - 2009-04-06 01:45:28
delta t: 2
heart - DEBUG chopped trace: Trace (IC, BJT, 0, Z)
timerange: 2009-04-06 01:44:04 - 2009-04-06 01:44:52
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982291.0
b: 1238982296.0
c: 1238982346.0
d: 1238982361.0
heart - DEBUG trace: Trace (IC, SSE, 0, Z)
timerange: 2009-04-06 01:44:40 - 2009-04-06 01:46:20
delta t: 2
heart - DEBUG chopped trace: Trace (IC, SSE, 0, Z)
timerange: 2009-04-06 01:44:56 - 2009-04-06 01:45:44
delta t: 2
heart - DEBUG taper times: --- !pf.CosTaper
a: 1238982251.0
b: 1238982256.0
c: 1238982306.0
d: 1238982321.0
heart - DEBUG trace: Trace (IU, CHTO, 0, Z)
timerange: 2009-04-06 01:44:00 - 2009-04-06 01:45:40
delta t: 2
heart - DEBUG chopped trace: Trace (IU, CHTO, 0, Z)
timerange: 2009-04-06 01:44:16 - 2009-04-06 01:45:04
delta t: 2
heart - DEBUG Post-process time 0.023910
heart - DEBUG Stack traces time 0.000000
heart - DEBUG Returning...
seismic - DEBUG Teleseismic forward model on test model takes: 0.840477
models - INFO Model building was successful!
models - INFO Using "bin" backend to store samples!
models - INFO ... Initiate Metropolis for Parallel Tempering...
proposal_distribution: Normal, tune_interval=10, n_chains=5
models.base - INFO ... Starting Parallel Tempering ...
distributed - INFO MPI is installed!
distributed - INFO Detected python3
distributed - INFO Done initialising mpi runner
distributed - INFO sampler directory: /home/lab/bayesiano/beat/beat/sampler/pt.py
distributed - DEBUG Running mpi with arguments: -n 6 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
Traceback (most recent call last):
File "/home/lab/anaconda3/envs/beat/bin/beat", line 11, in
load_entry_point('beat==1.0', 'console_scripts', 'beat')()
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 1686, in main
globals()'command_' + command
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/apps/beat.py", line 775, in command_sample
sample(step, problem)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/models/base.py", line 196, in sample
record_worker_chains=pa.record_worker_chains)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/pt.py", line 836, in pt_sample
project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 226, in run_mpi_sampler
samplerdir, n_jobs=n_jobs, loglevel=loglevel, project_dir=project_dir)
File "/home/lab/anaconda3/envs/beat/lib/python3.6/site-packages/beat-1.0-py3.6-linux-x86_64.egg/beat/sampler/distributed.py", line 158, in run
in the directory %s'''.lstrip() % (program, self.tempdir))
beat.sampler.distributed.MPIError: mpiexec has been invoked as "mpiexec"
in the directory /tmp/mpiexec-173lp6ia
distributed - WARNING not removing temporary directory: /tmp/mpiexec-173lp6ia
------------------------------------------------------------*-
6) Now running the line:
mpiexec -n 6 python3 /home/lab/bayesiano/beat/beat/sampler/pt.py debug /home/lab/Laquila_dc
in the temporary directory: /tmp/mpiexec-173lp6ia
I did not get any error and mpiexec started to execute the process.
------------------------------------------------------------*-
NOTE: I am working in a CPU with 6 cores and 12 threads, the mpi version is 4.0.3
Thanks
The text was updated successfully, but these errors were encountered: