-
Notifications
You must be signed in to change notification settings - Fork 26
/
concept
executable file
·2784 lines (2706 loc) · 96.2 KB
/
concept
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#!/usr/bin/env bash
# This file is part of CO𝘕CEPT, the cosmological 𝘕-body code in Python.
# Copyright © 2015–2024 Jeppe Mosgaard Dakin.
#
# CO𝘕CEPT is free software: You can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# CO𝘕CEPT is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with CO𝘕CEPT. If not, see https://www.gnu.org/licenses/
#
# The author of CO𝘕CEPT can be contacted at dakin(at)phys.au.dk
# The latest version of CO𝘕CEPT is available at
# https://github.com/jmd-dk/concept/
# This script runs the CO𝘕CEPT code.
# Run the script with the -h option to get help.
# Unless this file is being sourced,
# automatically export all variables when set.
being_sourced="True"
if [ "${BASH_SOURCE[0]}" == "${0}" ]; then
being_sourced="False"
fi
if [ "${being_sourced}" == "False" ]; then
set -a
fi
# If this file is being sourced, backups of 'this_file' and 'this_dir'
# are needed not to alter the values of these variables.
this_file_backup="${this_file}"
this_dir_backup="${this_dir}"
# Absolute paths to this file and its directory
this_file="$(readlink -f "${BASH_SOURCE[0]}")"
this_dir="$(dirname "${this_file}")"
# The user's current working directory
if [ -z "${workdir}" ]; then
workdir="$(pwd)"
fi
# For the terminal to be able to print Unicode characters correctly,
# we need to use a UTF-8 locale.
set_locale() {
# This function will set the locale through the LC_ALL and LANG
# environment variables. We want to use a supported UTF-8 locale.
# The preference order is as follows:
# en_US.UTF-8
# en_*.UTF-8
# C.UTF-8
# POSIX.UTF-8
# *.UTF-8
# We consider the suffix (UTF-8) valid regardless of the case and
# presence of the dash.
# Get all available locals.
locales="$(locale -a 2>/dev/null || :)"
if [ -z "${locales}" ]; then
return
fi
# Look for available UTF-8 locale
for prefix in "en_US" "en_*" "C" "POSIX" "*"; do
for suffix in "UTF-8" "UTF8" "utf-8" "utf8"; do
pattern="${prefix}.${suffix}"
for loc in ${locales}; do
if [[ "${loc}" == ${pattern} ]]; then
export LC_ALL="${loc}"
export LANG="${loc}"
return
fi
done
done
done
}
set_locale
# Set the terminal if unset or broken
if [ -z "${TERM}" ] || [ "${TERM}" == "dumb" ]; then
export TERM="linux"
fi
# ANSI/VT100 escape sequences
esc="\x1b"
# Text formatting
esc_normal="${esc}[0m"
esc_bold="${esc}[1m"
esc_italic="${esc}[3m"
esc_no_italic="${esc}[23m"
esc_red="${esc}[91m"
# Cursor movement
esc_up="${esc}[1A"
esc_erase="${esc}[K"
# The name of the program, nicely typeset
if [ -z "${esc_concept}" ]; then
esc_concept="CO${esc_italic}N${esc_no_italic}CEPT"
else
esc_concept="${esc_concept//\$\{esc_italic\}/${esc_italic}}"
esc_concept="${esc_concept//\$\{esc_no_italic\}/${esc_no_italic}}"
fi
# Enable extended globbing, used for negative wildcards !(...)
shopt -s extglob
# Load paths from the .path file
curr="${this_dir}"
while :; do
if [ -f "${curr}/.path" ]; then
source "${curr}/.path"
break
fi
if [ "${curr}" == "/" ]; then
# Print out error message and exit
printf "${esc_bold}${esc_red}Could not find the .path file!${esc_normal}\n" >&2
exit 1
fi
curr="$(dirname "${curr}")"
done
# Load environment variables from the .env file
env_backup_vars=(mpi_executor make_jobs)
env_backup_vals=()
for env_var in "${env_backup_vars[@]}"; do
eval "env_val=\"\${${env_var}}\""
env_backup_vals=("${env_backup_vals[@]}" "${env_val}")
done
source "${env}"
for ((index=0; index<${#env_backup_vars[@]}; index+=1)); do
env_val="${env_backup_vals[${index}]}"
if [ -n "${env_val}" ]; then
env_var="${env_backup_vars[${index}]}"
eval "${env_var}=\"${env_val}\""
env_backups=("${env_backups[@]}" "${env_val}")
fi
done
# Add the src directory to searched paths
# when importing modules in Python.
export PYTHONPATH="${src_dir}:${PYTHONPATH}"
# Disable Python hash randomization (salting) for reproducibility
export PYTHONHASHSEED=0
# Do not produce .pyc files
export PYTHONDONTWRITEBYTECODE=1
# Some Python packages may need access to libraries at runtime
for lib in "blas" "fftw" "gsl" "hdf5" "ncurses" "python" "zlib"; do
eval "lib_dir=\"\${${lib}_dir}\""
if [ -z "${lib_dir}" ]; then
continue
fi
lib_dir="${lib_dir}/lib"
if [ ! -d "${lib_dir}" ]; then
continue
fi
export LD_LIBRARY_PATH="${lib_dir}:${LD_LIBRARY_PATH}"
done
# A bug in HDF5 makes the code crash on certain file systems.
# A workaround is to set an environment variable as below.
export HDF5_USE_FILE_LOCKING=FALSE
# The MPI executables and libraries should be
# on the PATH and LD_LIBRARY_PATH, respectively.
export PATH="${mpi_bindir}:${PATH}"
if [ "$(basename "${mpi_libdir}")" == "merged_lib_specified" ]; then
if [ -d "${mpi_symlinkdir}" ]; then
# Let mpi_symlinkdir go before merged_lib_specified
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${mpi_symlinkdir}"
fi
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${mpi_libdir}"
else
export LD_LIBRARY_PATH="${mpi_libdir}:${LD_LIBRARY_PATH}"
fi
# Additional symlinks to MPI libraries might be placed in mpi_symlinkdir
if [ -d "${mpi_symlinkdir}" ]; then
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${mpi_symlinkdir}"
# If the special symlink named "ld_preload.so" is present
# in mpi_symlinkdir, we should include this symlink in LD_PRELOAD.
if [ -f "${mpi_symlinkdir}/ld_preload.so" ]; then
if [ -n "${LD_PRELOAD}" ]; then
export LD_PRELOAD="${LD_PRELOAD} ${mpi_symlinkdir}/ld_preload.so"
else
export LD_PRELOAD="${mpi_symlinkdir}/ld_preload.so"
fi
fi
fi
# The time before any computation begins.
# This time is saved both in seconds after the Unix epoch
# and in a human readable format.
start_time="$("${python}" -c "
import datetime
start_time_epoch = datetime.datetime.now().timestamp()
start_time_human = str(datetime.datetime.fromtimestamp(start_time_epoch))[:-3]
print(f'{start_time_epoch}_{start_time_human}')
")"
start_time_epoch="${start_time%%_*}"
start_time_human="${start_time#*_}"
# Further transform the human readable version
start_time_human_sec="${start_time_human%.*}"
start_time_human_nosep="${start_time_human//-/}"
start_time_human_nosep="${start_time_human_nosep// /}"
start_time_human_nosep="${start_time_human_nosep//:/}"
start_time_human_nosep="${start_time_human_nosep//./}"
# Check whether this script is run locally or remotely via ssh
ssh="True"
if [ -z "${SSH_CLIENT}" ] && [ -z "${SSH_TTY}" ]; then
ssh="False"
fi
# Function for printing coloured messages
colorprint() {
# Arguments: Message, colour
"${python}" -c "
import sys
from blessings import Terminal
terminal = Terminal(force_styling=True)
print(terminal.bold_${2}('${1}'), file=(sys.stderr if '${2}' == 'red' else sys.stdout))"
}
# Function for printing out a nice CO𝘕CEPT logo
print_logo() {
logo='
____ ____ __ ____ _____ ____ _____
/ __ \ / __ \ /\ / / / __ \ | ___| | _ \ |_ _|
| / \_| | / \ | / \ / / | / \_| | |__ | |_) | | |
|| _ || || / /\ \/ / || _ | __| | __/ | |
| \__/ | | \__/ | / / \ / | \__/ | | |___ | | | |
\____/ \____/ /_/ \/ \____/ |_____| |_| |_|
'
# Plot the logo via Python's matplotlib.
# While the colour of the 𝘕 is fixed, the colour used for the rest
# is determined from the timestamp of this file.
# This uses up the 16th and 17th colour of the terminal.
"${python}" -c "
import sys, matplotlib
# Generate colour based on input number
color_for_N = 'darkorange'
brightness = lambda color: 0.241*color[0]**2 + 0.691*color[1]**2 + 0.068*color[2]**2
blim = (0.3, 0.97)
color_choices = [
matplotlib.colors.ColorConverter().to_rgb(color)
for name, color in
matplotlib.colors.CSS4_COLORS.items()
if name != color_for_N
]
color_choices = [color for color in color_choices if blim[0] < brightness(color) < blim[1]]
colors = (
color_choices[int(sys.argv[1]) % len(color_choices)],
color_for_N,
)
# Apply colormap
for i, color in enumerate(colors):
colorhex = matplotlib.colors.rgb2hex(color)
print('\\x1b]4;{};rgb:{}/{}/{}\\x1b\\\\'
.format(16 + i, colorhex[1:3], colorhex[3:5], colorhex[5:]), end='')
# Construct the coloured logo
logo=r'''${logo}'''
logo = logo[1:-1]
rows = logo.split('\\n')
ANSI = []
for row in rows:
ANSI.append('\\x1b[1m')
for i, c in enumerate(row):
if i in {0, 18, 31}:
colornumber = 17 if i == 18 else 16
ANSI.append(f'\\x1b[38;5;{colornumber}m')
ANSI.append(c)
ANSI.append('\\x1b[0m\\n')
# Print the ANSI image
print(''.join(ANSI), end='', flush=True)
" \
$(stat -c '%Y' "${this_file}")
}
# Print out the logo the first time an execution reaches this point.
# Do not print the logo if the file is beind sourced or if the version
# should be printed.
if [ "${logo_printed}" != "True" ] \
&& [ "${being_sourced}" == "False" ] \
&& [[ " $@ " != *" -v " ]] \
&& [[ " $@ " != *" --v " ]] \
&& [[ " $@ " != *" --ve " ]] \
&& [[ " $@ " != *" --ver " ]] \
&& [[ " $@ " != *" --vers " ]] \
&& [[ " $@ " != *" --versi " ]] \
&& [[ " $@ " != *" --versio " ]] \
&& [[ " $@ " != *" --version " ]] \
; then
print_logo
export logo_printed="True"
fi
# Function for converting paths to absolute paths
absolute_path() {
# Arguments: Path, [working directory]
local path="${1}"
local current_dir="$(pwd)"
if [[ "${path}" == /* ]]; then
# The path is already absolute
echo "${path}"
return
elif [ -n "${2}" ]; then
# Explicit working directory supplied
cd "${2}"
elif [[ "${path}" == "./" ]] || [[ "${path}" == "../" ]]; then
# The path is explicitly written as relative
# to the user's current directory.
cd "${workdir}"
elif [ -f "${workdir}/${path}" ] || [ -d "${workdir}/${path}" ]; then
# Path exists relative to the user's current working directory
cd "${workdir}"
elif [ -f "${concept_dir}/${path}" ] || [ -d "${concept_dir}/${path}" ]; then
# Path exists relative to the concept directory
cd "${concept_dir}"
else
# Assume the path is relative to the user's current working directory
cd "${workdir}"
fi
# Place backslashes before spaces, dollar signs and parentheses.
# These are needed when expanding tilde, but they will not persist.
path="${path// /\\ }"
path="${path//$/\\$}"
path="${path//\(/\\(}"
path="${path//\)/\\)}"
# Expand tilde
eval path="${path}"
# Convert to absolute path
path="$(readlink -m "${path}")"
cd "${current_dir}"
if [ -z "${path}" ]; then
colorprint "Cannot convert \"${1}\" to an absolute path!" "red"
exit 1
fi
# Print out result
echo "${path}"
}
# Function for converting an absolute path to its "sensible" form.
# That is, this function returns the relative path with respect to the
# concept directory, if it is no more than one directory above the
# concept directory. Otherwise, return the absolute path back again.
sensible_path() {
"${python}" -c "
path = '${1}'
from os.path import relpath
rel = relpath(path, '${concept_dir}')
print(path if rel.startswith('../../../') else rel)"
}
# Function which prints the absolute path of a given command.
# If the command is not an executable file on the PATH but instead a
# known function, the input command is printed as is. If the command
# cannot be found at all, nothing is printed and an exit code of 1
# is returned.
get_command() {
command_name="${1}"
# Use the type built-in to locate the command
local path="$(type "${command_name}" 2>/dev/null | awk '{print $NF}')"
if [[ "${path}" == "/"* ]]; then
# The command is a path
path="$(readlink -f "${path}")"
echo "${path}"
return 0
elif [ -n "${path}" ]; then
# The command exists as a function
echo "${command_name}"
return 0
fi
# The command does not exist
return 1
}
# Function which prints a passed Bash array
# in the format of a Python list.
bash_array2python_list() {
# Call like this: bash_array2python_list "${array[@]}"
local list=''
local element
for element in "$@"; do
# If element is a string, encapsulate it in quotation marks
element=$("${python}" -c "
try:
eval(\"${element}\")
print(\"${element}\")
except Exception:
print('\"{}\"'.format(\"${element}\"))
")
# Append element to list
list="$(echo "${list}")${element}, "
done
list="[$(echo "${list}")]"
echo "${list}"
}
# Function for recursively removing empty sub-directories
# within the tmp directory (including the tmp directly itself).
cleanup_empty_tmp() {
local d
if [ ! -d "${tmp_dir}" ]; then
return
fi
tmp_subdirs=("${tmp_dir}")
deleted_any="False"
while :; do
tmp_subdirs_new=()
for tmp_subdir in "${tmp_subdirs[@]}"; do
for d in "${tmp_subdir}/"*; do
if [ ! -d "${d}" ]; then
continue
fi
is_empty="True"
for f in "${d}/"*; do
if [ "${f}" != "${d}/*" ]; then
# Much faster than ls -A
is_empty="False"
break
fi
done
if [ "${is_empty}" == "True" ]; then
rm -rf "${d}" || :
deleted_any="True"
else
tmp_subdirs_new=("${tmp_subdirs_new[@]}" "${d}")
fi
done
done
if [ ${#tmp_subdirs_new} -eq 0 ]; then
break
fi
tmp_subdirs=("${tmp_subdirs_new[@]}")
done
if [ "${deleted_any}" == "True" ]; then
if [ -z "$(ls -A "${tmp_dir}")" ]; then
rm -rf "${tmp_dir}" || :
else
cleanup_empty_tmp
fi
fi
}
# Determine the MPI implementation and version
if [ -f "${mpi_bindir}/ompi_info" ] || [ -f "${mpi_compilerdir}/ompi_info" ]; then
# OpenMPI
mpi_implementation="openmpi"
mpi_version="$("${mpiexec}" --version 2>&1 | head -n 1 | awk '{print $NF}')"
elif [ -f "${mpi_bindir}/mpichversion" ] || [ -f "${mpi_compilerdir}/mpichversion" ]; then
# MPICH or MVAPICH
for f in "${mpi_bindir}/mpichversion" "${mpi_compilerdir}/mpichversion"; do
if [ -f "${f}" ]; then
line="$("${mpi_bindir}/mpichversion" 2>&1 | head -n 1)"
mpi_implementation="$(echo "${line}" | awk '{print $1}' | tr '[:upper:]' '[:lower:]')"
if [[ "${mpi_implementation}" == "mvapich"* ]]; then
mpi_implementation="mvapich"
else
mpi_implementation="mpich"
fi
mpi_version="$(echo "${line}" | awk '{print $NF}')"
break
fi
done
else
# Unknown MPI implementation
mpi_implementation="unknown"
mpi_version="unknown"
fi
# MPI implementation specifics.
# While we always construct the contents of mpi_env,
# that of mpiexec_args is only constructed when this script is run
# as opposed to being sourced. We do this as mpiexec_args is only used
# when executing the program, and its construction requires test running
# mpiexec, which may cause trouble on some clusters.
if [ "${being_sourced}" == "False" ]; then
mpiexec_args=""
fi
mpi_env=""
if [ "${mpi_implementation}" == "openmpi" ]; then
# Disable aggregation of OpenMPI warnings
mpi_env="${mpi_env}
export OMPI_MCA_orte_base_help_aggregate=0"
# Disable OpenMPI warning about forking
mpi_env="${mpi_env}
export OMPI_MCA_mpi_warn_on_fork=0"
# By default, OpenMPI disallows any usage by the root user
mpi_env="${mpi_env}
export OMPI_ALLOW_RUN_AS_ROOT=1
export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1"
# In OpenMPI 3 and 4, oversubscription (having more MPI processes
# than physical cores) is disallowed by default.
mpi_version_major="${mpi_version:0:1}"
if [ "${mpi_version_major}" -ge 3 2>/dev/null ]; then
mpi_env="${mpi_env}
export OMPI_MCA_rmaps_base_oversubscribe=1"
fi
# In OpenMPI 4, InfiniBand ports are disabled by default in favour
# of UCX. If this leads to an error about not initializing an
# OpenFabrics device, we overwrite this default.
if [ -z "${OMPI_MCA_btl_openib_allow_ib}" ] \
&& [ "${mpi_version_major}" -ge 4 2>/dev/null ]; then
# As we are importing MPI4Py we ought to invoke Python via
# mpiexec, though this may lead to trouble when sourcing.
if [ "${being_sourced}" == "False" ]; then
mpi_warning="$("${mpiexec}" -n 1 "${python}" -c '
import mpi4py.rc; mpi4py.rc.threads = False # Do not use threads
from mpi4py import MPI
' 2>&1 || :)"
else
mpi_warning="$("${python}" -c '
import mpi4py.rc; mpi4py.rc.threads = False # Do not use threads
from mpi4py import MPI
' 2>&1 || :)"
fi
if [ -n "${mpi_warning}" ]; then
if echo "${mpi_warning}" | grep "error" | grep "OpenFabrics" >/dev/null; then
if echo "${mpi_warning}" | grep "btl_openib_allow_ib" >/dev/null; then
mpi_env="${mpi_env}
export OMPI_MCA_btl_openib_allow_ib=1"
fi
fi
fi
fi
# Disable automatic process binding/affinity, allowing OpenMP
# threads to be assigned to cores in a one-to-one fashion.
# This is off by default prior to OpenMPI version 1.7.
# All of 1.7 -- 4.0 supports the --bind-to none option to mpiexec.
# Here we add this option if it is understood.
if [ "${being_sourced}" == "False" ]; then
mpiexec_output="$("${mpiexec}" --bind-to none -n 1 echo "success" 2>&1 || :)"
if [ "${mpiexec_output}" == "success" ]; then
mpiexec_args="${mpiexec_args} --bind-to none"
fi
fi
# For OpenMPI 1.8 -- 4.0 process binding can also be deactivated
# using the following environment variable.
mpi_env="${mpi_env}
export OMPI_MCA_hwloc_base_binding_policy=none"
elif [ "${mpi_implementation}" == "mpich" ]; then
# Though MPICH should not use process binding by default,
# it still supports a -bind-to none or --bind-to none option.
# Here we check if this is so, and apply it.
if [ "${being_sourced}" == "False" ]; then
mpiexec_output="$("${mpiexec}" -bind-to none -n 1 echo "success" 2>&1 || :)"
if [ "${mpiexec_output}" == "success" ]; then
mpiexec_args="${mpiexec_args} -bind-to none"
else
mpiexec_output="$("${mpiexec}" --bind-to none -n 1 echo "success" 2>&1 || :)"
if [ "${mpiexec_output}" == "success" ]; then
mpiexec_args="${mpiexec_args} --bind-to none"
fi
fi
fi
elif [ "${mpi_implementation}" == "mvapich" ]; then
# Disable automatic process binding, allowing OpenMP threads to be
# assigned to cores in a one-to-one fashion.
mpi_env="${mpi_env}
export MV2_ENABLE_AFFINITY=0"
fi
if [ -n "${mpi_env}" ]; then
eval "${mpi_env}"
mpi_env="# Environment variables${mpi_env}"
fi
# Default values of command-line arguments
build_default="${build_dir}"
command_line_params_default=""
interactive_default="False"
job_directive_default=""
job_name_default="concept"
linktime_optimizations_default="True"
local_default="False"
main_default="${src_dir}/main.py"
memory_default="-1" # -1 implies unset
native_optimizations_default="False"
nprocs_default="1"
optimizations_default="True"
param_default="None" # None implies unset
pure_python_default="False"
queue_default=""
rebuild_default="" # empty implies that a rebuild is triggered by changes to the source
safe_build_default="True"
submit_default="" # empty implies submission if other remote job options are specified
tests_default=""
utility_default=""
walltime_default="00:00:00" # 00:00:00 implies unset
watch_default="True"
# Functions for building the code
set_make_jobs() {
# Arguments: [--force].
# Set the "make_jobs" variable, holding the -j option for
# future make commands, enabling parallel building.
# Some implementations of make do not support the bare -j
# option without explicitly specifying a number afterwards.
# If so, we do not make use of the -j option.
if [ "$1" != "--force" ] && ([ -n "${make_jobs}" ] || [ "${ssh}" == "True" ]); then
# Do not change the value of make_jobs
return
fi
tmp_dir_preexists="False"
if [ -d "${tmp_dir}" ]; then
tmp_dir_preexists="True"
fi
make_jobs_test_dir="${tmp_dir}/make_jobs_test_${start_time_human_nosep}"
rm -rf "${make_jobs_test_dir}" 2>/dev/null || :
mkdir -p "${make_jobs_test_dir}"
printf "
test:
\t@echo success
" > "${make_jobs_test_dir}/Makefile"
make_jobs_output="$(cd "${make_jobs_test_dir}" && make -j 2>/dev/null)" || :
if [ "${make_jobs_output}" == "success" ]; then
make_jobs="-j"
fi
if [ "${tmp_dir_preexists}" == "True" ]; then
rm -rf "${make_jobs_test_dir}" 2>/dev/null || :
else
rm -rf "${tmp_dir}" 2>/dev/null || :
fi
}
check_rebuild_necessary() {
# The code should be rebuild if the compiled modules are
# missing or out-of-date with respect to the source.
# The makefiles check this already, but we would like to
# avoid invoking make unnecessarily, as make is rather slow
# due to the large environment inherited by this Bash process.
source_files=("${src_dir}/"* "${concept_dir}/Makefile")
t_src=$( \
stat -c '%Y' \
"${source_files[@]}" \
| awk '(FNR==1){x=$1} {x=$1 > x?$1:x} END {print x}' \
)
t_build=$( \
stat -c '%Y' \
"${build}/"*.so \
2>/dev/null \
| awk '(FNR==1){x=$1} {x=$1 < x?$1:x} END {print x}' \
)
for f_src in "${source_files[@]}"; do
if [[ "${f_src}" == *.py ]]; then
f_base="$(basename "${f_src}")"
if [ "${f_base}" == "pyxpp.py" ]; then
continue
fi
f_build="${build}/${f_base%.py}.so"
if [ ! -s "${f_build}" ]; then
# Flag rebuild
t_build=""
break
fi
fi
done
if [ -z "${t_src}" ] || [ -z "${t_build}" ] || [ "${t_build}" -le "${t_src}" ]; then
# The build is not up-to-date
echo "True"
else
echo "False"
fi
}
build_concept() {
# In case of pure Python mode, run directly off of the Python source
if [ "${pure_python}" == "True" ]; then
build="${src_dir}"
# Note that ${src_dir} is already included in PYTHONPATH
return
fi
# Add the selected build directory as the first searched path
# when importing modules in Python.
export PYTHONPATH="${build}:${PYTHONPATH}"
# Check whether the code should be rebuild
if [ "${rebuild}" == "True" ]; then
rm -rf "${build}/"* "${build}/".[^.]*
elif [ "${rebuild}" == "False" ]; then
return
else
rebuild_actual="$(check_rebuild_necessary)"
if [ "${rebuild_actual}" == "False" ]; then
return
fi
fi
# Rebuild the code.
# As this process requires a lot of memory it may fail,
# in particular on virtual machines with dynamic memory
# and a small base memory. Attempting the compilation
# a few times in a row can help, as the dynamic memory
# then has a chance to increase.
start_time_build=$("${python}" -c "import time; print(time.time())")
set_make_jobs
make_concept() {
make \
build="${build}" \
optimizations="${optimizations}" \
linktime_optimizations="${linktime_optimizations}" \
native_optimizations="${native_optimizations}" \
safe_build="${safe_build}" \
${make_jobs}
}
n_make_attemps=3
for ((i = 0; i < ${n_make_attemps}; i += 1)); do
exec 4>&1
make_output="$(
cd "${concept_dir}" \
&& make_concept \
| tee >(cat >&4) \
; echo "concept_exit_code = ${PIPESTATUS[0]}" \
)"
exec 4>&-
exit_code="$(echo "${make_output}" | grep "concept_exit_code" \
| awk '{print $NF}' || :)"
if [ "${exit_code}" == "0" ]; then
break
fi
sleep 1
done
if [ "${exit_code}" != "0" ]; then
colorprint "Failed to compile ${esc_concept}!" "red"
exit 1
fi
if [[ "${make_output}" == *"Building modules"* ]]; then
"${mpiexec}" -n 1 "${python}" -c "from commons import *
print('Build time: {}'.format(time_since(${start_time_build})))"
fi
}
# Function for setting variables used
# when running CO𝘕CEPT via Python.
prepare_python_options() {
# Prepare Python options
if [ "${main_as_command}" == "True" ]; then
# Run main as Python command
main_as_library="${main}"
m_flag="-c"
else
if [ "${pure_python}" == "True" ]; then
# Run main as normal Python script
main_as_library="${main}"
m_flag=""
else
main_as_library="${main%.*}.so"
if [ -f "${main_as_library}" ]; then
# Run main as compiled library module
main_as_library="$(basename "${main_as_library}")"
m_flag="-m"
else
# Run main as normal Python script,
# even though the CO𝘕CEPT modules are compiled.
main_as_library="${main}"
m_flag=""
fi
fi
fi
i_flag=""
if [ "${interactive}" == "True" ]; then
i_flag="-i"
fi
}
# Function for printing basic information about a job
print_info() {
echo "Date: ${start_time_human_sec}" \
| tee -a "${job_dir}/${jobid}/log"
if [ -n "${job_name}" ] && [ "${job_name}" != "${job_name_default}" ]; then
echo "Job name: ${job_name}" \
| tee -a "${job_dir}/${jobid}/log"
fi
echo "Job ID: ${jobid}" \
| tee -a "${job_dir}/${jobid}/log"
if [ "${pure_python}" == "False" ] && [ "${build}" != "${build_dir}" ]; then
echo "Build: \"$(sensible_path "${build}")\"" \
| tee -a "${job_dir}/${jobid}/log"
fi
if [ "${main_as_command}" == "False" ] \
&& [[ "${main}" != "${src_dir}/main."* ]] \
&& [[ "${main}" != "${build}/main."* ]] \
; then
echo "Entry point: \"$(sensible_path "${main}")\"" \
| tee -a "${job_dir}/${jobid}/log"
fi
if [ "${param}" == "${param_default}" ]; then
echo "Parameter file: ${param}" \
| tee -a "${job_dir}/${jobid}/log"
else
echo "Parameter file: \"$(sensible_path "${param}")\"" \
| tee -a "${job_dir}/${jobid}/log"
fi
if [ -n "${memory_display}" ]; then
echo "Memory: ${memory_display}" \
| tee -a "${job_dir}/${jobid}/log"
fi
if [ -n "${walltime_display}" ]; then
echo "Wall time: ${walltime_display}" \
| tee -a "${job_dir}/${jobid}/log"
fi
}
# If this file is being sourced, return now
if [ "${being_sourced}" == "True" ]; then
this_file="${this_file_backup}"
this_dir="${this_dir_backup}"
return
fi
# Set up error trapping
ctrl_c() {
trap : 0
exit 2
}
abort() {
exit_code_newest=$?
colorprint "An error occurred!" "red"
if [ -n "${exit_code}" ]; then
exit ${exit_code}
elif [ ${exit_code_newest} -ne 0 ]; then
exit ${exit_code_newest}
else
exit 1
fi
}
trap 'ctrl_c' SIGINT
trap 'abort' EXIT
set -e
# Function which prints the resource manager.
# In order, the implemented resource managers are:
# - Slurm
# - TORQUE/PBS
get_resource_manager() {
# Detect what resource manager is used
if get_command sbatch >/dev/null; then
# Slurm is installed. Use this as the resource manager.
resource_manager="slurm"
elif get_command qsub >/dev/null; then
# TORQUE/PBS is installed. Use this as the resource manager.
resource_manager="torque"
else
# No resource manager found
resource_manager=""
fi
echo "${resource_manager}"
}
# Change to the concept directory
cd "${concept_dir}"
# Use Python's argparse module to handle command-line arguments
argparse_finished="False"
argparse_exit_code=""
args=$("${python}" -c "
import argparse, math, os, re, sys
# Function which checks whether input is a representation of
# one or two positive integers. If two ints are given,
# separate them by a colon.
def positive_int_or_int_pair(value, value_input=None):
value = str(value)
if value_input is None:
value_input = value
def raise_argparse_exception():
raise argparse.ArgumentTypeError(
f\"invalid positive int or int pair: '{value_input}'\"
)
for sep in ',;':
value = value.replace(sep, ':')
if value == '' or value.count(':') > 1:
raise_argparse_exception()
elif value.count(':') == 1:
values = value.split(':')
return ':'.join(positive_int_or_int_pair(value, value_input) for value in values)
else: # value.count(':') == 0
try:
value_eval = eval(value)
except Exception:
return positive_int_or_int_pair(value.replace(' ', ':'), value_input)
try:
value_float = float(value_eval)
except Exception:
raise_argparse_exception()
value_int = int(value_float)
if value_int == value_float and value_int > 0:
return str(value_int)
raise_argparse_exception()
# Function which checks whether input is a representation of
# a memory size and converts it to bytes.
def memory(value):
value = str(value)
value_raw = value
def raise_argparse_exception():
raise argparse.ArgumentTypeError(f\"invalid memory value: '{value_raw}'\")
# Convert to (whole) bytes
value = value.lower()
value = value.replace(' ', '').replace('b', '')
value = re.subn(r'([0-9]+)([a-z]+)', r'\g<1>*\g<2>', value)[0]
units = {
'k': 2**10,
'm': 2**20,
'g': 2**30,
't': 2**40,
'p': 2**50,
'e': 2**60,
'z': 2**70,
'y': 2**80,
}
try:
value = int(math.ceil(float(eval(value, units))))
except Exception:
raise_argparse_exception()
return value
# Function which converts a time value to the format hh:mm:ss
def time(value):
value = str(value)
# Convert value to integer seconds
units = {'s': 1}
units['sec'] = units['secs'] = units['seond'] = units['seonds'] = units['s']
units['m'] = 60*units['s']
units['min'] = units['mins'] = units['minute'] = units['minutes'] = units['m']
units['h'] = 60*units['m']
units['hr'] = units['hrs'] = units['hs'] = units['hour'] = units['hours'] = units['h']
units['d'] = 24*units['h']
units['day'] = units['days'] = units['d']
units['y'] = 365.25*units['d']
units['yr'] = units['year'] = units['years'] = units['y']
# If a pure number is provided, interpret this in seconds
value = value.replace(' ', '')
try:
value = int(value)*units['s']
except Exception:
pass
# Attempt to interpret the time as an expression
# like '2hr + 30mins'.
if isinstance(value, str):
value = value.lower()
value = re.subn(r'([0-9]+)([a-z]+)', r'\g<1>*\g<2>', value)[0]
try:
value = int(math.ceil(float(eval(value, units))))
except Exception:
pass
# Attempt to interpret the time in the format
# 'm:s' or 'h:m:s' or 'd:h:m:s' or 'd+h:m:s' or 'd+h:m' or 'd+h'.
if isinstance(value, str):
plusses_in_value = value.count('+')
if plusses_in_value > 1:
raise argparse.ArgumentTypeError(\"error parsing value\")
for sep in '+ ,;':
value = value.replace(sep, ':')
value = value.split(':')
s = m = h = d = 0
if len(value) == 1:
raise argparse.ArgumentTypeError(\"error parsing value\")
elif len(value) == 2:
if plusses_in_value:
# Format d:h
d, h = value
else:
# Format m:s
m, s = value
elif len(value) == 3:
if plusses_in_value:
# Format d+h:m
d, h, m = value
else:
# Format h:m:s
h, m, s = value
elif len(value) == 4:
# Format d:h:m:s or d+h:m:s
d, h, m, s = value
else:
raise argparse.ArgumentTypeError(\"error parsing value\")
d, h, m, s = int(d), int(h), int(m), int(s)
value = int(math.ceil(s*units['s'] + m*units['m'] + h*units['h'] + d*units['d']))
# Now value should be in integer seconds
if value < 0:
raise argparse.ArgumentTypeError(\"the wall time cannot be negative\")
# Convert to the format hh:mm:ss.
h = value//units['h']
value -= h*units['h']
m = value//units['m']
value -= m*units['m']
s = value
h, m, s = str(h), str(m), str(s)
if len(h) == 1:
h = '0' + h
if len(m) == 1:
m = '0' + m
if len(s) == 1:
s = '0' + s
value = f'{h}:{m}:{s}'
return value
# Function which checks whether input is a Boolean
def bool_from_str(value):
value_raw = value
value = str(value).lower()
if value.startswith('y') or value in {'true', 't', 'on', 'enable', '1', '1.0'}: