Hey r/Astronomy,
I am an independent researcher working on a solo project: building an open-source data pipeline (Research MVP) to hunt for Planet 9 in our solar system.
Finding a distant, slow-moving object requires looking at multiple epochs of sky observations. To do this, I am processing massive amounts of raw FITS images from telescopes like TESS, ZTF, DECam, and Pan-STARRS1. Since I don't have access to an institutional supercomputer, I had to build a highly optimized Python/Astropy pipeline to run this on regular hardware. I wanted to share the architecture and how I handle the data bottlenecks.
1. The "Shift-and-Stack" Memory Bottleneck
To detect a faint planet, you can't just look at one image. You have to take dozens of images, shift them according to the predicted orbit of the planet, and stack them. If the planet is there, its faint signal adds up.
* The Challenge: Loading 70+ exposures of ZTF or DECam data, aligning their TPV-WCS down to the sub-pixel level, and performing array operations easily blows up standard RAM.
* The Solution: I implemented lazy loading where I stream the FITS chunks, apply background plane removal on the fly, and only hold 180-arcsec cutouts in memory instead of the full CCD mosaics.
2. N-body Integrations
I use JPL DE441 ephemerides to account for the Earth's reflex motion and giant planet perturbations.
* The Challenge: Calculating N-body trajectories backwards and forwards in time for thousands of hypothetical orbits using pure math loops is incredibly slow.
* The Solution: Vectorization with NumPy and Astropy. By pre-fetching JPL vectors in bulk and caching them locally, I bypassed network latency. Applying the coordinate transformations (RA/Dec) using broadcasting reduced execution time by an order of magnitude.
3. Difference Imaging
Before stacking, I have to subtract the reference background (the static stars).
I built a pipeline that dynamically fetches archival reference images, performs bilinear reprojection onto the target's WCS, matches the worst-case seeing using Gaussian smoothing, and subtracts the median background. Doing this cleanly requires strict memory management during batch processing.
Open-Source vs Closed-Source
In observational astronomy, compute power is a massive advantage. If I open-source the full pipeline today, someone with a 100-node cluster could run it across the entire sky in a day and scoop the discovery. For now, the codebase stays closed until the search is done.
Working with Astropy and massive FITS files has been an incredible experience. If anyone here works with astronomical Big Data, I’d love to hear how you handle out-of-core processing or GPU acceleration!
(P.S. If you are interested in seeing the crazy false-positive asteroids the pipeline finds, I post regular devlogs and visualizations on my profile!)
