Data Access
Your portal to the Caterpillar data assets.
Halo catalogues and z = 0 snapshots are now available via Google cloud.
Storing the quantity of data produced by the Caterpillar pipeline is not without its headaches. Whilst the core data is stored on MIT computing infrastructure, the high value assets have been made available on Google Cloud infrastructure which allows for robust access for the lowest overhead. The only requirement is that in order for you to access all of the Caterpillar data, is that you install Google's Cloud Storage command line tools (gsutil).
Once installed, to list what is available in our bucket, you simply type:
$ gsutil ls -l gs://caterpillarproject/halos/
For more information on basic
gsutil usage, please see Google's quickstart documentation. A key function we've made to make this easier is as follows:import subprocess
def download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False):
cmd_list = []
for lx in lxs:
for snapshot in snapshots:
if want_halos:
stdout = subprocess.check_output("gsutil ls -d gs://caterpillarproject/halos/H*/H*LX%s*/halos_bound/halos_%s/" % (lx,snapshot),shell=True).decode("utf-8")
cmd_list.extend(["gsutil cp -r %s %s/%s" % (stouti,output_dir,"/".join(stouti.split("/")[3:])) for stouti in stdout.split()])
if want_particles:
stdout = subprocess.check_output("gsutil ls -d gs://caterpillarproject/halos/H*/H*LX%s*/outputs/snapdir_%s/" % (lx,snapshot),shell=True).decode("utf-8")
cmd_list.extend(["gsutil cp -r %s %s/%s" % (stouti,output_dir,"/".join(stouti.split("/")[3:])) for stouti in stdout.split()])
for cmdi in cmd_list:
subprocess.call([cmdi],shell=True)
See the redshift-snapshot key below to work out which snapshots you actually need. For example, to obtain the halos for snapshot 319 (z = 0), you simply type in your
ipython console and the download should begin.download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False)
A halo suite in the bucket is identified by its ID. e.g.
H1387186. These IDs are the Rockstar IDs from the parent simulation. To link actual Caterpillar numbers (from papers), use the following reference:
Name | PID | Name | PID | Name | PID | Name | PID |
Cat-1 | 1631506 | Cat-7 | 94687 | Cat-13 | 1725272 | Cat-19 | 1292085 |
Cat-2 | 264569 | Cat-8 | 1130025 | Cat-14 | 1195448 | Cat-20 | 95289 |
Cat-3 | 1725139 | Cat-9 | 1387186 | Cat-15 | 1599988 | Cat-21 | 1232164 |
Cat-4 | 447649 | Cat-10 | 581180 | Cat-16 | 796175 | Cat-22 | 1422331 |
Cat-5 | 5320 | Cat-11 | 1725372 | Cat-17 | 388476 | Cat-23 | 196589 |
Cat-6 | 581141 | Cat-12 | 1354437 | Cat-18 | 1079897 | Cat-24 | 1268839 |
Use the following legend to determine the parameters of the run:
H1387186_EB_Z127_P7_LN7_LX12_O4_NV4
H1387186 # halo rockstar id from parent simulation
EB # initial conditions type, 'B' for box" etc.
Z127 # starting redshift (z = 127)
P7 # padding parameters (2^7)^3
LN7 # level_min used in MUSIC (2^7)^3
LX12 # level_max used in MUSIC (2^12)^3
O4 # overlap parameter (2^4)^3
NV4 # number of times the virial radius enclosed defining lagrangian volume
e.g. all the available assets for a single halo at the highest resolution would be as follows:
halos/H1387186/H1387186_EB_Z127_P7_LN7_LX14_O4_NV4/
All halos were run at LX11, LX12, LX13 and LX14 resolutions with one done at LX15 to z = 1. For more details on these parameters, see the contamination suite information.
A given directory may have the following components:
H1387186_EB_Z127_P7_LN7_LX14_O4_NV4
-> halos_bound/ # rockstar and merger tree catalogues
-> halos_0/ # each folder contains the catalogue for each snapshot
-> halos_1/
...
-> halos_319/
-> outputs/
-> trees/
-> forests.list
-> locations.dat
-> tree_0_0_0.dat
-> tree_0_0_1.dat
...
-> tree_1_1_1.dat
-> tree.bin
-> treeindex.csv
-> outputs/ # gadget raw snapshot output (particle data)
-> snapdir_000/ # each folder contains the particle data for each snapshot
-> snapdir_001/
...
-> snapdir_319/
-> groups_319/ # the subfind catalogues are also stored (mostly for the last snapshot)
-> hsmldir_319/ # the smoothing lengths for the corresponding particle data
-> analysis/ # post-processed output files (halo profiles, mass functions, minihalos etc.)
As you can see above, in this directory, you'll find both halo catalogues (e.g. out.list files),
halos_bound/halos_[snapshot]/ # rockstar out.list files
and particle snapshots (e.g. HDF5 files).
outputs/snapdir_[snapshot]/ # Gadget HDF5 files
Lastly, we have a key to link redshift and snapshot (for those available):
Snapshot | Approximate Redshift |
319 | 0.000 |
232 | 0.501 |
189 | 0.996 |
145 | 2.011 |
124 | 2.975 |
111 | 3.959 |
102 | 4.984 |
95 | 6.002 |
81 | 7.006 |
70 | 8.023 |
62 | 8.941 |
54 | 10.066 |
43 | 12.108 |
32 | 15.073 |
21 | 19.771 |
The full expansion factor list can be found at the following two links:
ExpansionList.txt
4KB
Text
Expansion Factor List (LX14)
ExpansionList_256.txt
3KB
Text
Expansion Factor List (<LX14), 256 snapshot runs
Gentle reminder that the expansion factor, a = 1/(1+redshift), the index of the expansion factor in the above files is the snapshot number (it is 0 indexed, e.g. the first row in the expansion list file is the the expansion factor of snapshot_000, so for snapshot 0, the redshift is... 1/0.021276596 - 1 ~46).
Once you have
gsutil installed, to obtain all the Caterpillar ROCKSTAR catalogues (and retain the directory structure), simply use:$ gsutil cp gs://caterpillarproject/halos/H*/H*/halos_bound/ ./
Alternatively, if you would like a specific Caterpillar halo's catalogues, use the function:
download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False)
Using
gsutil, again you can obtain a certain halo's Gadget HDF5 snapshot (z = 0) via;download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False)
Last modified 3yr ago