-
Kepler & K2 Science Conference V Poster
This past week, I was at the Kepler and K2 Science Conference, held in Glendale California. This conference allowed me to interact with a myriad of amazingly intelligent people doing amazing research in many different areas of astronomy and cosmology, and I can capitalize on this with further research. While I have quite a bit I'd like to say about my time at the conference, I'd like to use this post to share the poster I presented at the conference, which focused on searching for exoplanets in dense clusters.
(Note: Google Drive seems to automatically compress PDF files in their preview application; for those interested in seeing the poster in full quality, one can download it by opening the file in another window and looking in the upper right hand corner) -
A Basic Sedov Simulation Animation
Using FLASH, I was able to run a two dimensional simulation of the Sedov Explosion. By setting the program to record data at about 300 points throughout the explosion, I was able to generate and save a yt slice plot of each checkpoint and then compile these in Adobe Premiere to make a gif of the explosion. The result should be displayed below
One particularly interesting part of this visualization is the ability to see how FLASH is able to automatically increase the resolution on the area of interest, the explosion, to simulate it more accurately as needed without using excess computing power in areas were there isn't anything relevant. This adaptive mesh grid refinement FLASH uses is an extremely powerful tool for research and is necessary for the research in astrophysics that I assist in.
-
Creating a FLASH Site-Specific Makefile for the Dark Cluster
The Making of a Makefile
I recently identified a potential way to improve our process for running new simulations on FLASH on the Dark Cluster. While it isn't a high priority, I felt it would be a good exercise to familiarize myself with navigating using the terminal and customizing FLASH. I found it enlightening to document the issues I had and my eventual solution, hopefully for the benefit of others if this issue is encountered
Standard Procedure
When running a new simulation on the Dark Cluster using the latest version of FLASH, version 4.5, the process is fairly straightforward, but it most certainly could be optimized. After signing in, tools for compiling are loaded using the command module load intel/15.0.1 && module load intelmpi/5.0.2.044 . To save time on login, I have this command in my bash.rc file in my root directory, which runs on login. Once in the flash directory, one moves to flash/object and uses the command make clean to ensure that any files and links from the prior simulation are cleared. Moving back into flash , one initializes the desired simulation using the command ./setup [Simulation] -[Flags]. Some common flags used are -3d and -auto, which set the simulation to be run in 3 dimensions and automatically select the correct Makefile respectively. As an aside, the simulation being run is located in the directory flash/source/Simulation/SimulationMain with all of the necessary data to run the simulation located within a folder with the name of the simulation. I will likely explore the creation of a new complex simulation in a future blog post.
This point is where I saw the potential for improving the efficiency of our process. Currently, we change into the object directory, delete the automatically generated Makefile.h and replace it with a preconfigured file that links all the correct dependencies and their directories. This is obviously not the most efficient means of accomplishing this, and there is potential for contributing to FLASH by adding the Dark Cluster to the list of sites with a preconfigured Makefile. I will elaborate on this further, but, for now, I will finish describing the process.
After adding the correct Makefile.h, one runs make in the flash/object directory, which compiles the simulation. However, I find it is much quicker to run this with the flag -j [threads], with [threads] being the number of threads the compilation is run on. I found 8 threads was more than sufficient to compile the simulation in a minute or so.
At this point, there should be a file called flash.par located in flash/object. This is copied outside of the flash folder into a new folder. It's useful to title this the name of the simulation. After this, the bash file with the parameters for how the simulation should be run on the cluster (I call this jobScript.sh but the name isn't particularly important) is copied into the same folder. Changing directories to this folder, running the command sbatch jobScript.sh will start the simulation, and the files of the results should be placed in this folder once the simulation is done.A Note on Other Systems
Most of this procedure is the same on any Unix-based system, with some minor deviations. While the last paragraph is partially specific to the process used when working on the Dark Cluster, the basic process is likely similar on other systems, albeit with different syntax and a different way to initiate the simulation. The published documentation for a given system is usually enough to figure out how to run the simulation.A Potential Improvement
When initializing FLASH, particularly when using the -auto flag, FLASH finds the hostname of the computer and compares it against a list of systems that have preconfigured Makefiles contained within the FLASH package. This list can be found in the directory flash/sites. Also located here is the file Aliases , which contains a list of sites that report themselves with a hostname other than the one that can be used to connect to the site. For example, entering the hostname command while on the first Dark Cluster server returns the name fend01.cluster while the host name used to connect to the cluster is fend01.hpc.ku.dk
As of this writing, the Dark Cluster is not included in this list of sites. Because of this, when FLASH attempts to initialize the setup script, it attempts to identify and use the Makefile it believes is most suitable for the system it is running on. When running the command ./setup Sedov -3d -auto , or any ./setup file for that matter, FLASH uses the Makefile located in the directory flash/sites/fen.bluegenebnl.gov which does not specify the correct directories for FLASH dependencies.
When attempting to continue the process of compiling a simulation using this Makefile, compilation in the flash/object directory with the make command fails because FLASH is unable to access the necessary MPI compilers that allow it to compile Fortran 95 code, which FLASH is largely written in. This is problematic because there is no clear indication that the Makefile is listing incorrect directories (in fact, FLASH actually prints SUCCESS upon initializing even if the Makefile is incorrect).
Adding a new set of folders to the flash/sites directory for the Dark Cluster would be beneficial to reduce the potential points of failure in the process of compiling a simulation, but my attempts to do so have proved challenging.
Attempts at Execution
First off, there appears to be some disparity between the hostname reported when using the hostname command and the hostname used to connect to the Dark Cluster. The command returns fend01.cluster while the actual hostname used to connect is fend01.hpc.ku.dk . This is actually addressed by FLASH, which stores the file Aliases in the flash/sites directory. This file is used to make the hostname reported by the system an alias of the actual hostname. This allows FLASH to initialize using the correct preconfigured Makefile even if the hostname reported isn't the actual hostname. However, the Dark Cluster is not present in this file, and, as stated before, it is not one of the sites with a preconfigured Makefile.
When attempting to add a preconfigured Makefile, I ran into a few interesting issues that I hope can be enlightening. After adding a directory to flash/sites with the name fend01.hpc.ku.dk , I then added the Makefile provided by Aldo Batta in his extremely helpful guide on using FLASH on the Dark Cluster. Upon running ./setup Sedov -auto , FLASH defaulted to the fen.bluegenebnl.gov Makefile. Changing the directory name of the Dark Cluster Makefile to fend01.cluster yielded the same result, and adding the reported and correct hostnames to the Aliases file ended in the same outcome. Out of curiosity, I removed the line in Aliases that references fen.bluegenebnl.gov in an attempt to force FLASH to use the inserted Makefile. Upon doing this, FLASH used the inserted Dark Cluster Makefile and a basic Sedov simulation was able to run and I was able to verify that the data was useable in yt, the Python library we use for most analysis of hdf5 files.
A basic slice plot of the temperature of the Sedov explosion towards the end of the simulation
I found it odd that FLASH was using the correct Makefile when the alias for fen.bluegenebnl.gov was removed. It was at this point that I noticed two things- I had placed my aliases for the Dark Cluster at the bottom of the list, and the alias for fen.bluegenebnl.gov was fen . Upon restoring the alias for fen.bluegenebnl.gov and moving the aliases I had set for the Dark Cluster above this alias, FLASH began to consistently use my inserted Makefile with the -auto flag.
Conclusions
From what I can gather, FLASH searches the Aliases file for the first alias that contains part of the reported hostname. Because the alias for fen.bluegenebnl.gov was fen, when FLASH searched for a match to fend01.cluster , it interpreted fen to be the correct alias because it is the first three letters of the reported hostname. upon moving the correct alias above this one, FLASH was able to match the reported alias correctly. While this may not be an issue that breaks the program, it certainly can be a minor inconvenience. Though I am currently not familiar enough with Fortran and the inner workings of the FLASH program to fix the underlying issue myself, this workaround is functional for my uses and likely will be for others.
While this write up was certainly longer than it had to be to document this, I found it useful to commit to words the actual procedure of compiling the simulation and the process I went through while solving this issue. Admittedly, this post went into quite a bit of detail on these subjects, but hopefully it can be useful for others using FLASH and to look back on should trouble arise in the future. -
A 3-Dimensional View of a Star Cluster Volume Render
While I haven't yet progressed to the point of simulating my own globular clusters in FLASH (hopefully soon), I was fortunate enough to be supplied with a hdf5 checkpoint of a complex simulation so I could practice volume rendering in yt. This was supplied by Melinda Soares-Furtado, whose website can be found by clicking on her name. After getting the hang of volume rendering fairly quickly thanks to the fairly straightforward syntax of Python and the yt library, I decided to try my hand at creating a visualization that imparted a sense of depth. While yt has an experimental interactive volume renderer included at the time of this writing (more information about this can be found here), I wanted to try my hand at creating a gif, which doesn't need to be rendered in real time whenever I wanted to show it.
With some help from Melinda. who introduced me to some useful python commands that allowed me to name the files in an orderly manner, I was able to create a fairly pleasing visualization. I created the frames by using a for loop that created a volume render of the density data in the file, saved the image, and then rotated the camera around the center of the cluster by one degree. I had this repeat until each degree was rendered. From there, I compiled them into a sequence in Adobe Premiere and rendered a gif.
Unfortunately, the sigma_clip parameter, which increases the gamma value of the function to make the visualization easier to see, for the yt function show() seems to only apply to the image shown in the jupyter notebook file and not the file that is downloaded with the save() function. The original gif is shown below
With this visualization, it's fairly difficult to make out the cluster at all, and it isn't particularly useful. Thankfully, I was able to import it back into Adobe Premiere and make increase the saturation and gamma, along with some other color correction, to make the visualization much more intriguing
In the future, I hope to create a similar visualization that shows a simulation actually running as the camera rotates around the scene rather than capturing an instant in time. I think this would be very interesting, and it would undoubtedly be a way to increase my proficiency in using yt for data visualization and analysis. I also plan on investigating the issue of the more saturated render not being downloaded by yt, as it really wouldn't be practical to manually save each render preview instead of using the save() function.
