# Introduction ## Welcome to the documentation for Caterpillar. Here you can find information pertaining to the following: * Historical design and implementation decisions. * Data products. * How to obtain access. * Basic data manipulation using public tools. * Authorship policies. {% hint style="info" %} The status of The Caterpillar Project is now effectively **Public.** {% endhint %} If you require any help or support, head to the [community Slack channel](https://caterpillarproject.slack.com). # Available Data ## Particle Data The core N-body code used to run Caterpillar was a combination of `P-Gadget3` and `Gadget-4`. The output of the simulation runs is the typical Gadget HDF5 output and should be compatible with all other downstream post-processing tools available in the community. These outputs are available at all 320 snapshots of the simulation from z = 127 to z = 0 with < 50 Myr resolution from z = 6 to z = 0. For details, please see either our flagship paper or our Technical pages. {% hint style="info" %} Owing to the extreme size of the runs, particle data (z > 0) is only available upon request. {% endhint %} ## Halo Catalogues {% tabs %} {% tab title="Rockstar" %} We used a modified version of `ROCKSTAR` included full iterative unbinding but the outputs are consistent with the nominal outputs from standard `ROCKSTAR` catalogues. Please see the documentation at the [Rockstar repository](https://bitbucket.org/gfcstanford/rockstar/src/master/README.md) for details (See [Section 4. Outputs](https://bitbucket.org/gfcstanford/rockstar/src/master/README.md#markdown-header-output)). There are however a few important caveats relating to our outputs which differ from the nominal outputs. e.g. M\_grav vs. Mvir vs npart\*m\_p {% endtab %} {% tab title="SUBFIND" %} These can be made available upon request. {% endtab %} {% endtabs %} ## Merger Trees The chosen merger tree system used is [consistent-trees](https://bitbucket.org/pbehroozi/consistent-trees/src/master/). Please see the `consistent-trees` [README](https://bitbucket.org/pbehroozi/consistent-trees/src/master/README.md) for direct information pertaining to its running and output. # Access & Authorship Policies {% hint style="success" %} Halo catalogues, merger trees z = 0 snaphots are now **Public**. See "Data Access" for access. Whilst you do not need to consult the Caterpillar collaboration prior to publication, we kindly request you add general research project and contact details to the "Projects page" to potentially foster new collaborations of mutual interest. {% endhint %} {% hint style="warning" %} Owing to the size of the raw z > 0 particle data, it available upon request only. {% endhint %} ## Private (Particle) Data We first distinguish between two sets of collaboration members: * **Contributing members**: Members that have lead a project and/or have made specific concrete contributions to a project/paper. They should be added to papers according to their contributions. * **Core members**: Caterpillar Core members who initiated the Caterpillar project. They should be invited to join papers in \[alphabetical/predetermined/TBD] order. For any project by Contributors or Core members using particle data, given their past contributions to producing the Caterpillar data set, all Core members should automatically be offered co-authorship. {% hint style="danger" %} Near-ready versions of papers must be sent out to all Contributing authors and Core Members for comments **before** submission/publication, with a \~3 week response time. {% endhint %} Core members must actively accept being a co-author by communicating with the paper’s first author, and by providing some contribution to the paper via initial ideas and/or contributions to the project, comments on the draft, suggestions for improvement to text/figures etc, or others, within that time frame.\Once you're strong enough, save the world: # FAQ ## Should I contact a Caterpillar core member to use the data? For using the merger tree and halo catalogs, no, you now no longer need to contact a core member. That being said however, we do encourage you to contact a core member as certain efforts may have been undertaken by groups either internally and externally and we can put you in direct contact with them to coordinate efforts. ## Why aren't the particle snapshots public? This is primarily because unlink many other simulation suites, the particle resolution and snapshot resolution make it expensive to 100% host on public servers (e.g. Google Cloud Platform). Should you want more than the z = 0 snapshot, please get in touch so we can find the best path forward for your project. # Overview & Basic Approach ## Overview The general approach of Caterpillar was that of the zoom-in technique adopted by many other groups, albeit with a few differences. we performed a much, much larger search for optimal computer parameters to ensure that contamination volumes are as large as possible without great cost. We also implemented iterative unbinding in `ROCKSTAR` which we found to be critical in identifying halos at the highest resolution simulations, particularly on highly radial orbits. ![Caterpillar halos were drawn from a parent simulation and re-simulated at high resolution.](/files/-LeT6dFdnstBIBeUAiPz) ## Considerations ### Volume The volume of the parent simulation was selected to be 100 Mpc/h as this allows for roughly \~6500 Milky Way-sized (i.e. 10^12 Msol) systems to be found. After a gentle selection over local environment (i.e. making sure no halos were near clusters) 2122 candidates were used to select *Caterpillar* candidates. ### Mass Resolution We required a resolution which allowed us to resolve $$10^{12} M\_\odot$$ halos with 10,000 particles so as to construct well defined lagrangian volumes. This resulted in us selecting a resolution of 1024^3 or a particle mass of $$8.72 \times 10^{12} M\_\odot/h$$ . ### Halo Selection We selected halos with the following environmental requirements: * halos between 0.7 - 3 x 1012 \\(M\_:raw-latex:odot\\) (6564 candidates) * no halos larger than 7 x 1013 \\(M\_:raw-latex:odot\\) within 7 Mpc * no halos larger than 7 x 1012 \\(M\_:raw-latex:odot\\) within 2.8 Mpc (2122 candidates) This is roughly in line with Tollerud et al. (2012), Boylan-Kolchin et al. (2013), Fardal et al. (2013), Pfiffel et al. (2013), Li & White (2008), van der Marel et al. (2012), Karachentsev et al. (2004) and Tikhonov & Klypin (2009). This avoids Milky Way-sized systems near clusters but does not make them overly isolated necessarily. Halos were also selected to not be preferentially near the very edge of the simulation volume as a matter of convenience. The first 24 *Caterpillar* halos are highlighted within the parent volume below. ### Temporal Resolution The time steps were set to be log of the expansion factor, following a similar convention to that used by the *Millenium* and *Millenium-II* simulations. The following table shows the various measures for time/size at each snapshot. ## Halo Properties {% hint style="info" %} Nearly all of the following can be found in our flagship paper [Griffen et al. (2015)](https://iopscience.iop.org/article/10.3847/0004-637X/818/1/10/pdf). {% endhint %} ### Halo Profiles ![](/files/-LeT98E6qzWFBq0mFQqx) ### Accretion History Lower Resolution Runs ![](/files/-LeT944MyHmSvbmxVUko) ### Subhalos ![](/files/-LeT9BftFTd6EWcU1P43) ### Convergence ![](/files/-LeTC3FZKSLKJi6lq8oh) ![](/files/-LeT9mNgFu0tUjHB0ExQ) ### Mass-Concentration ![](/files/-LeTA1m70jKTSnlia5zd) ### Relaxed ![](/files/-LeT7FZXBR5y3t-jZ3eO) ### Lower Resolution Runs ![](/files/-LeT9XW_8403-p2oWMvg) ![](/files/-LeT9UUgnvQrtEHwME8s) # Initial Conditions ## Overview We use the multi-scale cosmological initial conditions creator `MUSIC`. `MUSIC` is a computer program to generate nested grid initial conditions for high-resolution “zoom” cosmological simulations. A detailed description of the algorithms can be found in Hahn & Abel (2011). You can download the user’s guide [here](https://bitbucket.org/ohahn/music/downloads/MUSIC_Users_Guide.pdf) or obtain a copy of the code [here](https://people.phys.ethz.ch/~hahn/MUSIC/). Any questions should be directed to [Brendan Griffen](mailto:brendan.f.griffen%40gmail.com) and then if that fails, [Oliver Hahn](mailto:hahn%40phys.ethz.ch). ## Parameters These are the parameter files which were used to generate the initial conditions for the *Caterpillar* project. We adopt the raw Planck (2013) cosmology and do not use 2nd order Lagrangian perturbation theory. The [user guide](https://bitbucket.org/ohahn/music/downloads/MUSIC_Users_Guide.pdf) contains all the information required to understand the following parameter files. Within each simulation directory, there is a file named `OUTPUTmusic` which gives the log of the construction of the initial conditions. # Parent Simulation ## Initial Conditions First we constructed a parent simulation of sufficient size to include thousands of Milky Way sized systems but also of sufficient resolution to construct good Lagrangian volumes. We decided on \~10,000 particles per Milky Way-sized host and a 100 Mpc/h volume. Our `level_max=10` , which means our parent simulation’s effective resolution is (2^10)^3 = 1024^3, i.e. a parent particle mass, $$ m\_p = 8.72 \times 10^7 M\_\odot $$ ``` # parentics.conf [setup] boxlength = 100 zstart = 127 levelmin = 10 levelmin_TF = 10 levelmax = 10 padding = 8 overlap = 4 ref_center = 0.5, 0.5, 0.5 ref_extent = 0.2, 0.2, 0.2 align_top = yes baryons = no use_2LPT = no use_LLA = no periodic_TF = yes [cosmology] Omega_m = 0.3175 Omega_L = 0.6825 Omega_b = 0.049 H0 = 67.11 sigma_8 = 0.8344 nspec = 0.9624 transfer = eisenstein [random] seed[10] = 34567 [output] format = gadget2 filename = ./ics gadget_num_files = 128 [poisson] fft_fine = yes accuracy = 1e-5 pre_smooth = 3 post_smooth = 3 smoother = gs laplace_order = 6 grad_order = 6 ``` # Zooming In Each simulation folder has a configuration file for MUSIC (e.g. `H1079897_EX_Z127_P7_LN7_LX14_O4_NV4.conf`) which was used to construct the initial conditions. There are a few things to note about the zoom-in parameter files when compared to the parent volume parameter file. First is we now specify a `region_point_file` which defines the x,y,z (normalized to the box width) of the particles to be re-sampled. We have added the parameter `hipadding` which our own modification to allow for expanded regions. 1.05, for example, represents an expanded ellipsoid (by 5%). See Section 2.3 of [Griffen et al. (2015)](http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:1509.01255) for a more detailed description of these geometries and their impact on contamination. We also draw the reader’s attention to the seed values. Note that we do not set the seed values for any level lower than the parent volume (10) which makes MUSIC smooth out any levels lower than 10. The seed values at 11 are simply the halo numbers and then each level higher scales by a factor of 2 of this original number. This was required because the ICs were generated through our automatic pipeline and each simulation needs unique values to seed the random noise field. The `levelmin_TF` is also set to be the same as the parent volume (10). The padding and overlap parameters are the same for all simulations. ## MUSIC Parameter File ``` # H1079897_EX_Z127_P7_LN7_LX14_O4_NV4.conf [setup] boxlength = 100 zstart = 127 levelmin = 7 levelmin_TF = 10 levelmax = 14 padding = 7 overlap = 4 region = ellipsoid hipadding = 1.05 region_point_file = /n/home01/bgriffen/data/caterpillar/ics/lagr/H1079897NRVIR4 align_top = no baryons = no use_2LPT = no use_2LLA = no periodic_TF = yes [cosmology] Omega_m = 0.3175 Omega_L = 0.6825 Omega_b = 0.049 H0 = 67.11 sigma_8 = 0.8344 nspec = 0.9624 transfer = eisenstein [random] seed[10] = 34567 seed[11] = 1079897 seed[12] = 2159794 seed[13] = 3239691 seed[14] = 4319588 [output] format = gadget2_double filename = ./ics gadget_num_files = 8 gadget_spreadcoarse = yes [poisson] fft_fine = yes accuracy = 1e-05 pre_smooth = 3 post_smooth = 3 smoother = gs laplace_order = 6 grad_order = 6 ``` #### Halo Selection We selected halos with the following environmental requirements: * halos mass between $$0.7 \leq M\_{vir} \leq 3 \times 10^{12} M\_\odot$$ (6564 candidates) * no halos larger than $$7 \times 10^{13} M\_\odot$$ within 7 Mpc * no halos larger than $$7 \times 10^{12} M\_\odot$$ within 2.8 Mpc (2122 candidates) This is roughly in line with Tollerud et al. (2012), Boylan-Kolchin et al. (2013), Fardal et al. (2013), Pfiffel et al. (2013), Li & White (2008), van der Marel et al. (2012), Karachentsev et al. (2004) and Tikhonov & Klypin (2009). This avoids Milky Way-sized systems near clusters but does not make them overly isolated necessarily. Halos were also selected to not be preferentially near the very edge of the simulation volume as a matter of convenience. The first 24 *Caterpillar* halos are highlighted within the parent volume below. #### Temporal Resolution The time steps were set to be log of the expansion factor, following a similar convention to that used by the *Millenium* and *Millenium-II* simulations. The following table shows the various measures for time/size at each snapshot. Be sure to use the halo utility module (`haloutils`) in Python for quickly getting the temporal quantity for a given snapshot. See data access for more information. | Snap | Scale Factor | Redshift | Time | | ---- | ------------ | -------- | ------- | | 0 | 0.0213 | 46.0000 | 0.0535 | | 1 | 0.0290 | 33.5029 | 0.0851 | | 2 | 0.0367 | 26.2557 | 0.1212 | | 3 | 0.0444 | 21.5245 | 0.1613 | | 4 | 0.0521 | 18.1929 | 0.2051 | | 5 | 0.0598 | 15.7199 | 0.2522 | | 6 | 0.0675 | 13.8114 | 0.3025 | | 7 | 0.0752 | 12.2940 | 0.3557 | | 8 | 0.0829 | 11.0586 | 0.4117 | | 9 | 0.0906 | 10.0333 | 0.4704 | | 10 | 0.0983 | 9.1687 | 0.5316 | | 11 | 0.1060 | 8.4297 | 0.5952 | | 12 | 0.1138 | 7.7909 | 0.6612 | | 13 | 0.1215 | 7.2331 | 0.7294 | | 14 | 0.1292 | 6.7419 | 0.7998 | | 15 | 0.1369 | 6.3060 | 0.8723 | | 16 | 0.1446 | 5.9166 | 0.9469 | | 17 | 0.1523 | 5.5666 | 1.0234 | | 18 | 0.1600 | 5.2503 | 1.1018 | | 19 | 0.1677 | 4.9630 | 1.1821 | | 20 | 0.1754 | 4.7011 | 1.2642 | | 21 | 0.1831 | 4.4611 | 1.3481 | | 22 | 0.1908 | 4.2406 | 1.4337 | | 23 | 0.1985 | 4.0371 | 1.5210 | | 24 | 0.2062 | 3.8489 | 1.6098 | | 25 | 0.2139 | 3.6742 | 1.7003 | | 26 | 0.2216 | 3.5117 | 1.7923 | | 27 | 0.2294 | 3.3601 | 1.8858 | | 28 | 0.2371 | 3.2184 | 1.9808 | | 29 | 0.2448 | 3.0856 | 2.0772 | | 30 | 0.2525 | 2.9608 | 2.1749 | | 31 | 0.2602 | 2.8435 | 2.2741 | | 32 | 0.2679 | 2.7330 | 2.3745 | | 33 | 0.2756 | 2.6286 | 2.4762 | | 34 | 0.2833 | 2.5299 | 2.5792 | | 35 | 0.2910 | 2.4364 | 2.6834 | | 36 | 0.2987 | 2.3477 | 2.7888 | | 37 | 0.3064 | 2.2635 | 2.8953 | | 38 | 0.3141 | 2.1835 | 3.0029 | | 39 | 0.3218 | 2.1072 | 3.1116 | | 40 | 0.3295 | 2.0346 | 3.2213 | | 41 | 0.3372 | 1.9652 | 3.3321 | | 42 | 0.3449 | 1.8990 | 3.4438 | | 43 | 0.3527 | 1.8356 | 3.5565 | | 44 | 0.3604 | 1.7750 | 3.6701 | | 45 | 0.3681 | 1.7169 | 3.7846 | | 46 | 0.3758 | 1.6612 | 3.9000 | | 47 | 0.3835 | 1.6077 | 4.0161 | | 48 | 0.3912 | 1.5563 | 4.1331 | | 49 | 0.3989 | 1.5069 | 4.2508 | | 50 | 0.4066 | 1.4594 | 4.3693 | | 51 | 0.4143 | 1.4137 | 4.4884 | | 52 | 0.4220 | 1.3696 | 4.6082 | | 53 | 0.4297 | 1.3271 | 4.7287 | | 54 | 0.4374 | 1.2861 | 4.8497 | | 55 | 0.4451 | 1.2465 | 4.9714 | | 56 | 0.4528 | 1.2083 | 5.0936 | | 57 | 0.4605 | 1.1713 | 5.2163 | | 58 | 0.4683 | 1.1356 | 5.3395 | | 59 | 0.4760 | 1.1010 | 5.4632 | | 60 | 0.4837 | 1.0675 | 5.5873 | | 61 | 0.4914 | 1.0351 | 5.7118 | | 62 | 0.4991 | 1.0037 | 5.8367 | | 63 | 0.5068 | 0.9732 | 5.9620 | | 64 | 0.5145 | 0.9437 | 6.0876 | | 65 | 0.5222 | 0.9150 | 6.2135 | | 66 | 0.5299 | 0.8871 | 6.3396 | | 67 | 0.5376 | 0.8601 | 6.4660 | | 68 | 0.5453 | 0.8338 | 6.5927 | | 69 | 0.5530 | 0.8082 | 6.7195 | | 70 | 0.5607 | 0.7834 | 6.8465 | | 71 | 0.5684 | 0.7592 | 6.9737 | | 72 | 0.5761 | 0.7357 | 7.1010 | | 73 | 0.5838 | 0.7128 | 7.2284 | | 74 | 0.5916 | 0.6905 | 7.3559 | | 75 | 0.5993 | 0.6687 | 7.4835 | | 76 | 0.6070 | 0.6475 | 7.6111 | | 77 | 0.6147 | 0.6269 | 7.7387 | | 78 | 0.6224 | 0.6067 | 7.8663 | | 79 | 0.6301 | 0.5871 | 7.9939 | | 80 | 0.6378 | 0.5679 | 8.1215 | | 81 | 0.6455 | 0.5492 | 8.2490 | | 82 | 0.6532 | 0.5309 | 8.3764 | | 83 | 0.6609 | 0.5131 | 8.5038 | | 84 | 0.6686 | 0.4956 | 8.6310 | | 85 | 0.6763 | 0.4786 | 8.7581 | | 86 | 0.6840 | 0.4619 | 8.8851 | | 87 | 0.6917 | 0.4456 | 9.0119 | | 88 | 0.6994 | 0.4297 | 9.1385 | | 89 | 0.7072 | 0.4141 | 9.2649 | | 90 | 0.7149 | 0.3989 | 9.3912 | | 91 | 0.7226 | 0.3840 | 9.5172 | | 92 | 0.7303 | 0.3694 | 9.6430 | | 93 | 0.7380 | 0.3551 | 9.7685 | | 94 | 0.7457 | 0.3410 | 9.8938 | | 95 | 0.7534 | 0.3273 | 10.0188 | | 96 | 0.7611 | 0.3139 | 10.1436 | | 97 | 0.7688 | 0.3007 | 10.2680 | | 98 | 0.7765 | 0.2878 | 10.3922 | | 99 | 0.7842 | 0.2752 | 10.5160 | | 100 | 0.7919 | 0.2627 | 10.6395 | | 101 | 0.7996 | 0.2506 | 10.7627 | | 102 | 0.8073 | 0.2386 | 10.8855 | | 103 | 0.8150 | 0.2269 | 11.0081 | | 104 | 0.8228 | 0.2154 | 11.1302 | | 105 | 0.8305 | 0.2042 | 11.2520 | | 106 | 0.8382 | 0.1931 | 11.3734 | | 107 | 0.8459 | 0.1822 | 11.4944 | | 108 | 0.8536 | 0.1715 | 11.6151 | | 109 | 0.8613 | 0.1611 | 11.7354 | | 110 | 0.8690 | 0.1508 | 11.8552 | | 111 | 0.8767 | 0.1406 | 11.9747 | | 112 | 0.8844 | 0.1307 | 12.0938 | | 113 | 0.8921 | 0.1209 | 12.2124 | | 114 | 0.8998 | 0.1113 | 12.3307 | | 115 | 0.9075 | 0.1019 | 12.4485 | | 116 | 0.9152 | 0.0926 | 12.5659 | | 117 | 0.9229 | 0.0835 | 12.6828 | | 118 | 0.9306 | 0.0745 | 12.7994 | | 119 | 0.9383 | 0.0657 | 12.9155 | | 120 | 0.9461 | 0.0570 | 13.0311 | | 121 | 0.9538 | 0.0485 | 13.1464 | | 122 | 0.9615 | 0.0401 | 13.2611 | | 123 | 0.9692 | 0.0318 | 13.3755 | | 124 | 0.9769 | 0.0237 | 13.4894 | | 125 | 0.9846 | 0.0157 | 13.6028 | | 126 | 0.9923 | 0.0078 | 13.7158 | | 127 | 1.0000 | 0.0000 | 13.8283 | The majority of the information about the zoom-in runs can be found in [Griffen et al. (2016)](http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:1509.01255). Here we simply outline some details which were left out of the publication for the sake of brevity. ## Resolution Levels | Aquarius Level | MUSIC `levelmax` | Effective Resolution | $$10^4 h^{-3} M\_\odot$$ | $$10^4 h^{-3} M\_\odot$$ | Force Softening $$\epsilon$$ (pc/h) | | -------------- | ---------------- | -------------------- | ------------------------ | ------------------------ | ----------------------------------- | | 1 | 15 | 32768^ 3 | 0.25 | 0.37 | 36 | | **2** | **14** | **1638 4^3** | **2** | **3** | **76** | | 3 | 13 | 8096^3 | 16 | 24 | 152 | | 4 | 12 | 4096^3 | 128 | 190 | 228 | | 5 | 11 | 2048^3 | 1025 | 1527 | 452 | Each panel represents one single realization of the Cat-1 halo at different resolutions. The far left is an `LX11` run and the far right is an `LX14` run. {% hint style="info" %} The LX15 run has currently only been run for one of the halos and has been temporarily paused at z = 1. This will be finished with a few others once the main suite has been completed. {% endhint %} We have complete (modified) `ROCKSTAR` halo catalogues (together with consistent-trees merger trees) and z = 0 `SUBFIND` catalogues. ## Force Softening Softening was 1/80th the inter-particle separation. We adopt the formula: `boxwidth/lx^2/80` but stagger the force softening for each higher level as 4 x base, 8 x base, 32 x base, 64 x base where base is the base force softening. For each of the zooms, this equates to (units of Mpc/h): | In Gadget | LX11 | LX12 | LX13 | LX14 | | ---------------- | ----------- | ----------- | ----------- | ------------ | | `SofteningHalo` | 0.000610352 | 0.000305176 | 0.000152588 | 0.0000762939 | | `SofteningDisk` | 0.002441406 | 0.001220703 | 0.000610352 | 0.000305176 | | `SofteningBulge` | 0.004882813 | 0.002441406 | 0.001220703 | 0.000610352 | | `SofteningStars` | 0.01953125 | 0.009765625 | 0.004882813 | 0.002441406 | | `SofteningBndry` | 0.0390625 | 0.01953125 | 0.009765625 | 0.004882813 | ## Temporal Resolution {% hint style="info" %} Timesteps are spaced logarithmically in expansion factor to z = 6, then linearly spaced in expansion factor down to z = 0. Always be aware of this as it could be strength and a weakness of your study. {% endhint %} This image shows the difference between the time step resolutions used in Caterpillar and those used in the Aquarius simulation. We wanted higher resolution at all redshifts for many purposes. At z > 6 we wanted to model Lyman-Werner radiation which requires timesteps of order the lifespan of Population III star formation. At low redshift we wanted timesteps of roughly 50-60 Myrs which is the disruption time scale of many small dwarf galaxies of the Milky Way. This also allows detailed modelling of the pericentric passages of infalling satellite systems, which is a crucial parameter for determining post-infall mass loss, for example. ## Contamination Study A number of contamination studies have been carried out. This involves changing the Lagrangian geometry in some way to keep the contamination (distance to the nearest particle type 2 as far as possible) low whilst conserving CPU hours. Our selected test geometries were as follows | Geometry | Details | | -------- | ----------------------------------------------------------------------------- | | BA | Original MUSIC bounding box (e.g . the exac t boun ding box of lagr volu me). | | BB | 1.2 bounding box extent | | BC | 1.4 bounding box extent | | BD | 1.6 bounding box extent | | CA | Convex Hull Volume | | EA | Original MUSIC Ellipsoid (e.g . the exact bounding box of Lagrangian volume). | | EB | 1.1 padding | | EC | 1.2 padding | | EX | 1.05 padding | We did this for both 4 and 5 times the virial radius at z = 0 (marked by the letter 4 or 5 at the end of the abbreviated geometry). Making a total of \~18 test halos per *Caterpillar* halo. Our requirement was that there was no contamination (particle type 2) within 1 Mpc of the host at the LX11 level. We also looked at how the geometry of the Lagrangian volume affected the contamination radius. As outlined in [Griffen et al. (2015)](http://adsabs.harvard.edu/cgi-bin/bib_query?arXiv:1509.01255), we did not find any correlation with geometry and overall level contamination. Every simulation requires its own tailored geometry to achieve our contamination requirements. The size of the Lagrangian volumes were also another challenge to overcome. If a halo had LX11 ICs which were larger than 300mb, we found that we could not run these at LX14 on national facilities. The size and distance became our two biggest obstacles when running the *Caterpillar* suite. Our `ROCKSTAR` catalogues only use the high-resolution particles. This means that there will be halos in the outskirts of the simulation which are contaminated. These are shown clearly below. Be sure not to just take all halos within the `ROCKSTAR` catalogues as some of them will be contaminated (underestimated masses, wrong profiles etc.). As a safety, one should only take halos which are within the contamination distance. This changes as a function of redshift so make sure you update your cut for each snapshot. The plots below are for z = 0. # Halo Identification # Merger Trees # Semi-Analytic Models # Data Access Your portal to the Caterpillar data assets. ## Google Cloud Platform {% hint style="success" %} Halo catalogues and z = 0 snapshots are now available via Google cloud. {% endhint %} Storing the quantity of data produced by the Caterpillar pipeline is not without its headaches. Whilst the core data is stored on MIT computing infrastructure, the high value assets have been made available on Google Cloud infrastructure which allows for robust access for the lowest overhead. The only requirement is that in order for you to access all of the Caterpillar data, is that you install Google's Cloud Storage command line tools ([gsutil](https://cloud.google.com/storage/docs/gsutil)). Please [install gsutil](https://cloud.google.com/storage/docs/gsutil_install) for your system. Once installed, to list what is available in our bucket, you simply type: ``` $ gsutil ls -l gs://caterpillarproject/halos/ ``` For more information on basic `gsutil` usage, please see [Google's quickstart documentation](https://cloud.google.com/storage/docs/quickstart-gsutil). A key function we've made to make this easier is as follows: ```python import subprocess def download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False): cmd_list = [] for lx in lxs: for snapshot in snapshots: if want_halos: stdout = subprocess.check_output("gsutil ls -d gs://caterpillarproject/halos/H*/H*LX%s*/halos_bound/halos_%s/" % (lx,snapshot),shell=True).decode("utf-8") cmd_list.extend(["gsutil cp -r %s %s/%s" % (stouti,output_dir,"/".join(stouti.split("/")[3:])) for stouti in stdout.split()]) if want_particles: stdout = subprocess.check_output("gsutil ls -d gs://caterpillarproject/halos/H*/H*LX%s*/outputs/snapdir_%s/" % (lx,snapshot),shell=True).decode("utf-8") cmd_list.extend(["gsutil cp -r %s %s/%s" % (stouti,output_dir,"/".join(stouti.split("/")[3:])) for stouti in stdout.split()]) for cmdi in cmd_list: subprocess.call([cmdi],shell=True) ``` See the redshift-snapshot key below to work out which snapshots you actually need. For example, to obtain the halos for snapshot 319 (z = 0), you simply type in your `ipython` console and the download should begin. ``` download_caterpillar(output_dir="./",lxs=[14],snapshots=[319],want_halos=True,want_particles=False) ``` Please talk to one of the team at [Caterpillar's Slack channel](https://caterpillarproject.slack.com/) should you have any problems. ## Conventions & Structures A halo suite in the bucket is identified by its ID. e.g. `H1387186`. These IDs are the Rockstar IDs from the parent simulation. ### Halo Key To link actual Caterpillar numbers (from papers), use the following reference: | Name | PID | Name | PID | Name | PID | Name | PID | | ----- | ------- | ------ | ------- | ------ | ------- | ------ | ------- | | Cat-1 | 1631506 | Cat-7 | 94687 | Cat-13 | 1725272 | Cat-19 | 1292085 | | Cat-2 | 264569 | Cat-8 | 1130025 | Cat-14 | 1195448 | Cat-20 | 95289 | | Cat-3 | 1725139 | Cat-9 | 1387186 | Cat-15 | 1599988 | Cat-21 | 1232164 | | Cat-4 | 447649 | Cat-10 | 581180 | Cat-16 | 796175 | Cat-22 | 1422331 | | Cat-5 | 5320 | Cat-11 | 1725372 | Cat-17 | 388476 | Cat-23 | 196589 | | Cat-6 | 581141 | Cat-12 | 1354437 | Cat-18 | 1079897 | Cat-24 | 1268839 | Use the following legend to determine the parameters of the run: ``` H1387186_EB_Z127_P7_LN7_LX12_O4_NV4 H1387186 # halo rockstar id from parent simulation EB # initial conditions type, 'B' for box" etc. Z127 # starting redshift (z = 127) P7 # padding parameters (2^7)^3 LN7 # level_min used in MUSIC (2^7)^3 LX12 # level_max used in MUSIC (2^12)^3 O4 # overlap parameter (2^4)^3 NV4 # number of times the virial radius enclosed defining lagrangian volume ``` e.g. all the available assets for a single halo at the highest resolution would be as follows: ``` halos/H1387186/H1387186_EB_Z127_P7_LN7_LX14_O4_NV4/ ``` All halos were run at LX11, LX12, LX13 and LX14 resolutions with one done at LX15 to z = 1. For more details on these parameters, see the contamination suite information. ### Folder Structure A given directory may have the following components: ``` H1387186_EB_Z127_P7_LN7_LX14_O4_NV4 -> halos_bound/ # rockstar and merger tree catalogues -> halos_0/ # each folder contains the catalogue for each snapshot -> halos_1/ ... -> halos_319/ -> outputs/ -> trees/ -> forests.list -> locations.dat -> tree_0_0_0.dat -> tree_0_0_1.dat ... -> tree_1_1_1.dat -> tree.bin -> treeindex.csv -> outputs/ # gadget raw snapshot output (particle data) -> snapdir_000/ # each folder contains the particle data for each snapshot -> snapdir_001/ ... -> snapdir_319/ -> groups_319/ # the subfind catalogues are also stored (mostly for the last snapshot) -> hsmldir_319/ # the smoothing lengths for the corresponding particle data -> analysis/ # post-processed output files (halo profiles, mass functions, minihalos etc.) ``` As you can see above, in this directory, you'll find both **halo catalogues** (e.g. out.list files), ``` halos_bound/halos_[snapshot]/ # rockstar out.list files ``` and **particle snapshots** (e.g. HDF5 files). ``` outputs/snapdir_[snapshot]/ # Gadget HDF5 files ``` ### Snapshot-Redshift Key Lastly, we have a key to **link redshift and snapshot** (for those available): | Snapshot | Approximate Redshift | | :------: | :------------------: | | 319 | 0.000 | | 232 | 0.501 | | 189 | 0.996 | | 145 | 2.011 | | 124 | 2.975 | | 111 | 3.959 | | 102 | 4.984 | | 95 | 6.002 | | 81 | 7.006 | | 70 | 8.023 | | 62 | 8.941 | | 54 | 10.066 | | 43 | 12.108 | | 32 | 15.073 | | 21 | 19.771 | The full expansion factor list can be found at the following two links: {% file src="/files/-MCjFLFtKRhUNh5COe38" %} Expansion Factor List (LX14) {% endfile %} {% file src="/files/-MCjFP6SXIQebQ08MmCR" %} Expansion Factor List (\ try tree. then tab complete to see other functions mainbranch = tree.getMainBranch() # we also have a short hand version: # mainbranch = htils.get_mainbranch(hpath) # which is very fast and skips the reading of the entire progenitor tree if you don't need dit # see further down this page for more options # output the main branch virial mass print mainbranch['mvir'] ``` Here is an example of some of these in action: ```python # load required modules import haloutils as htils import numpy as np # select Cat-1 halo hpaths = htils.get_paper_paths_lx(14)[0] # select the last snapshot (z = 0) snapshot = htils.get_lastsnap(hpath) # load rockstar id of the host halo zoomid = htils.load_zoomid(hpath) #Load Halo Catalogue halos = htils.load_rscat(hpath,snapshot) #Select host halos hosts = halos.get_hosts() #Select subhalos subs = halos.get_subs() # Get positions of subs and hosts print hosts[['posX','posY','posZ']] print subs[['posX','posY','posZ']] #Get particle ids from halo of interest (here it is the host) print halos.get_particles_from_halo(zoomid) #Get virial radius of a specific halo id (in this case the host) print halos.ix[zoomid]['rvir'] # units of kpc/h ``` ## Merger Trees Similarly the merger tree catalogues (once loaded) have a number of its own functions. ```python def getMainBranch(self, row=0): """ @param row: row of the halo you want the main branch for. Defaults to row 0 Uses getSubTree, then finds the smallest dfid that has no progenitors @return: all halos that are in the main branch of the halo specified by row (in a np structured array) """ def getMMP(self, row): """ @param row: row number (int) of halo considered @return: row number of the most massive parent, or None if no parent """ def getNonMMPprogenitors(self,row): """ return row index of all progenitors that are not the most massive These are all the subhalos that were destroyed in the interval """ ``` These can be used in the following example: ```python # load required modules import haloutils as htils import numpy as np # select the caterpillar halo of interest # based on halo id and resolution level hid = 1387186 lx = 14 hpath = htils.hid_hpath_lx(hid,lx) # select the last snapshot (z = 0) snapshot = htils.get_lastsnap(hpath) # load rockstar id of the host halo zoomid = htils.load_zoomid(hpath) # Load every tree (VERY slow) trees = htils.load_mtc(hpath) # Just look at the first tree # tree = cat[0] # You can access all trees via: cat[0], cat[1] etc. # If you want to load the tree for a particular ID from the rockstar catalogue trees = htils.load_mtc(haloids=[zoomid]) # in this case the host (quite slow) tree = cat[0] # tree will contain all progenitors, including subhalos # Just say you want to index the merger tree by the z = 0 root rockstar id # (i.e. the base of the tree). This is quite powerful because you might select # halos of interest in the rockstar catalogue then want to know, just for those # what their merger tree is (e.g. say you just want dwarf systems of a particular size) trees = htils.load_mtc(haloids=[zoomid],indexbyrsid=True) tree = trees[zoomid] # You can just loop through rockstar ids (if you gave it more than one id above) # and get out the accretion histories for a small sample of trees quite quickly # to get the main branch main_branch = tree.getMainBranch() # print mass evolution for mass,scale in zip(main_branch['mvir'],main_branch['scale']): print "%3.2f: %3.2e" % (scale,mass) # output 1.00: 2.86e+14 0.99: 2.89e+14 0.98: 2.90e+14 0.97: 2.90e+14 0.95: 2.88e+14 0.94: 2.86e+14 0.93: 2.83e+14 0.92: 2.80e+14 0.91: 2.76e+14 0.90: 2.73e+14 0.89: 2.64e+14 0.88: 2.47e+14 ... ``` A function to find the descendant branch of any halo in merger tree catalogue. You should use it as follows: ```python # get a tree of interest mtc = haloutils.load_zoom_mtc(hpath) host = mtc.Trees[0] # make a dictionary that maps ids to rows desc_map = host.get_desc_map() # get the descendent branch. desc_branch = host.getDescBranch(row, desc_map) # similarly for main branches, you can use a dictionary that speeds up the # getMainBranch call substantially. mmp_map = host.get_mmp_map() main_branch = host.getMainBranch(row, mmp_map) # you can get the branches without making the map as below, but they will be much slower. # only faster if you use ~ < 10 calls to the function. desc_branch = host.getDescBranch(row) main_branch = host.getMainBranch(row) ``` ## Particle Data If you want the Gadget header: ```python # get the first halo in the catalogue hpaths = htils.get_paper_paths_lx(14)[0] # read its Gadget header header = htils.get_halo_header(hpath) # header contains the typical Gadget info header.boxsize header.massarr header.omegaL header.cooling header.metals header.redshift header.double header.nall header.sfr header.feedback header.nall_highword header.stellar_age header.filenum header.npart header.time header.hubble header.omega0 ``` Be sure to divide the relevant quantities (pos, rvir etc.) by `header.hubble`. See the Gadget section in the sidebar for more information on the header and block types available. If you wanted to get the postions of all the particles for a specific halo (or any block). ```python pos = htils.load_partblock(hpath,zoomid,"POS ") # units Mpc/h # "VEL ", "ID ", "MASS" etc. also work (notice the space) # check readsnapshots/readsnapHDF5.py for the other block names # you can call in the caterpillar modules print pos*1000. # kpc/h # output [ 32085.45117188 57312.79296875 44314.35546875] [ 27002.18554688 10062.73242188 9899.70019531] [ 26711.08789062 9560.22460938 10165.18847656] [ 49757.3515625 21470.00195312 6461.90917969] ... ``` If you want to read in the entire block, use the following: ```python import haloutils as htils hid = 1387186 lx = 14 hpath = htils.hid_hpath_lx(hid,lx) pos = htils.load_partblock(hpath,319,"POS ") mass = htils.load_partblock(hpath,319,"MASS") ``` {% hint style="info" %} Note that the mass block will have different values depending on how many layers of refinement there are for that zoom in simulation. If you use this code on a parent simulation it will be an array of length N all of the same value because there is only one particle type. {% endhint %} If you wanted just the ids for a selection of particle ids: ```python pos = htils.load_partblock(hpath,zoomid,"POS ",partids=[listofids]) # units Mpc/h ``` # Help & Support ### Slack Channel We currently have a [Slack channel](https://caterpillarproject.slack.com) dedicated to cross-institute collaboration. This is your best option for getting an answer as soon as possible. ### Direct Email If Slack isn’t your thing, please contact the Project Lead, [Brendan Griffen via email](mailto:brendan.f.griffen%40gmail.com) at any time.