Data Access

Your portal to the Caterpillar data assets.

Google Cloud Platform

Halo catalogues and z = 0 snapshots are now available via Google cloud.
Storing the quantity of data produced by the Caterpillar pipeline is not without its headaches. Whilst the core data is stored on MIT computing infrastructure, the high value assets have been made available on Google Cloud infrastructure which allows for robust access for the lowest overhead. The only requirement is that in order for you to access all of the Caterpillar data, is that you install Google's Cloud Storage command line tools (gsutil).
Please install gsutil for your system.
Once installed, to list what is available in our bucket, you simply type:
$ gsutil ls -l gs://caterpillarproject/halos/
For more information on basic gsutil usage, please see Google's quickstart documentation. A key function we've made to make this easier is as follows:
import subprocess
def download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False):
cmd_list = []
for lx in lxs:
for snapshot in snapshots:
if want_halos:
stdout = subprocess.check_output("gsutil ls -d gs://caterpillarproject/halos/H*/H*LX%s*/halos_bound/halos_%s/" % (lx,snapshot),shell=True).decode("utf-8")
cmd_list.extend(["gsutil cp -r %s %s/%s" % (stouti,output_dir,"/".join(stouti.split("/")[3:])) for stouti in stdout.split()])
if want_particles:
stdout = subprocess.check_output("gsutil ls -d gs://caterpillarproject/halos/H*/H*LX%s*/outputs/snapdir_%s/" % (lx,snapshot),shell=True).decode("utf-8")
cmd_list.extend(["gsutil cp -r %s %s/%s" % (stouti,output_dir,"/".join(stouti.split("/")[3:])) for stouti in stdout.split()])
for cmdi in cmd_list:
subprocess.call([cmdi],shell=True)
See the redshift-snapshot key below to work out which snapshots you actually need. For example, to obtain the halos for snapshot 319 (z = 0), you simply type in your ipython console and the download should begin.
download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False)
Please talk to one of the team at Caterpillar's Slack channel should you have any problems.

Conventions & Structures

A halo suite in the bucket is identified by its ID. e.g. H1387186. These IDs are the Rockstar IDs from the parent simulation.

Halo Key

To link actual Caterpillar numbers (from papers), use the following reference:
Name
PID
Name
PID
Name
PID
Name
PID
Cat-1
1631506
Cat-7
94687
Cat-13
1725272
Cat-19
1292085
Cat-2
264569
Cat-8
1130025
Cat-14
1195448
Cat-20
95289
Cat-3
1725139
Cat-9
1387186
Cat-15
1599988
Cat-21
1232164
Cat-4
447649
Cat-10
581180
Cat-16
796175
Cat-22
1422331
Cat-5
5320
Cat-11
1725372
Cat-17
388476
Cat-23
196589
Cat-6
581141
Cat-12
1354437
Cat-18
1079897
Cat-24
1268839
Use the following legend to determine the parameters of the run:
H1387186_EB_Z127_P7_LN7_LX12_O4_NV4
H1387186 # halo rockstar id from parent simulation
EB # initial conditions type, 'B' for box" etc.
Z127 # starting redshift (z = 127)
P7 # padding parameters (2^7)^3
LN7 # level_min used in MUSIC (2^7)^3
LX12 # level_max used in MUSIC (2^12)^3
O4 # overlap parameter (2^4)^3
NV4 # number of times the virial radius enclosed defining lagrangian volume
e.g. all the available assets for a single halo at the highest resolution would be as follows:
halos/H1387186/H1387186_EB_Z127_P7_LN7_LX14_O4_NV4/
All halos were run at LX11, LX12, LX13 and LX14 resolutions with one done at LX15 to z = 1. For more details on these parameters, see the contamination suite information.

Folder Structure

A given directory may have the following components:
H1387186_EB_Z127_P7_LN7_LX14_O4_NV4
-> halos_bound/ # rockstar and merger tree catalogues
-> halos_0/ # each folder contains the catalogue for each snapshot
-> halos_1/
...
-> halos_319/
-> outputs/
-> trees/
-> forests.list
-> locations.dat
-> tree_0_0_0.dat
-> tree_0_0_1.dat
...
-> tree_1_1_1.dat
-> tree.bin
-> treeindex.csv
-> outputs/ # gadget raw snapshot output (particle data)
-> snapdir_000/ # each folder contains the particle data for each snapshot
-> snapdir_001/
...
-> snapdir_319/
-> groups_319/ # the subfind catalogues are also stored (mostly for the last snapshot)
-> hsmldir_319/ # the smoothing lengths for the corresponding particle data
-> analysis/ # post-processed output files (halo profiles, mass functions, minihalos etc.)
As you can see above, in this directory, you'll find both halo catalogues (e.g. out.list files),
halos_bound/halos_[snapshot]/ # rockstar out.list files
and particle snapshots (e.g. HDF5 files).
outputs/snapdir_[snapshot]/ # Gadget HDF5 files

Snapshot-Redshift Key

Lastly, we have a key to link redshift and snapshot (for those available):
Snapshot
Approximate Redshift
319
0.000
232
0.501
189
0.996
145
2.011
124
2.975
111
3.959
102
4.984
95
6.002
81
7.006
70
8.023
62
8.941
54
10.066
43
12.108
32
15.073
21
19.771
The full expansion factor list can be found at the following two links:
ExpansionList.txt
4KB
Text
Expansion Factor List (LX14)
ExpansionList_256.txt
3KB
Text
Expansion Factor List (<LX14), 256 snapshot runs
Gentle reminder that the expansion factor, a = 1/(1+redshift), the index of the expansion factor in the above files is the snapshot number (it is 0 indexed, e.g. the first row in the expansion list file is the the expansion factor of snapshot_000, so for snapshot 0, the redshift is... 1/0.021276596 - 1 ~46).

Obtaining Rockstar Halos

Once you have gsutil installed, to obtain all the Caterpillar ROCKSTAR catalogues (and retain the directory structure), simply use:
$ gsutil cp gs://caterpillarproject/halos/H*/H*/halos_bound/ ./
Alternatively, if you would like a specific Caterpillar halo's catalogues, use the function:
download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False)

Snapshot Particle Data (z = 0)

Using gsutil, again you can obtain a certain halo's Gadget HDF5 snapshot (z = 0) via;
download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False)