Openuniverse tutorial updates ahead of Sep 15 webinar by jkrick · Pull Request #370 · Caltech-IPAC/irsa-tutorials

jkrick · 2026-08-15T01:27:37Z

Updates the OpenUniverse2024 tutorials to the full data release and reconciles how they find data.
Closes #309.

What changed

All five notebooks use the full release, not the preview subset. Preview references are gone —
a reader has no reason to know an earlier subset existed.
Both telescopes are reached the same way. The firefly notebook built Roman paths from a hardcoded
24-value block grid while asking SIA for Rubin. Both now go through SIA, removing the grid constants,
block_wcs(), find_block(), get_roman_coadd_fpath(), and the grid-vs-header verification.
Every static figure is rotated to north up. Neither survey delivers north-up images (this Roman
exposure sits ~170° from north, the Rubin visit ~160°), so the four figures in §2/§3/§4/§6 were
previously all at different orientations. Each is now resampled onto a north-up grid keeping its own
pixel scale, so Roman's finer sampling stays visible. Whole exposures resample at coarser sampling for
display. Rotation leaves empty corners, so stretch_color() is nan-aware. This is what reproject is
for now; the old reproject_to_TAN() is gone.
The supernova notebook replaces a hardcoded instances CSV with an SIA positional query, and gains
playback controls and visit-coverage guidance.
The Roman coadds notebook is deleted. See below.
Reorganized landing page and toc: a nested OpenUniverse2024 group listed first, Quickstart at the
top, prefix-free titles, simulated.md split into two galleries.
Helper functions folded into hidden cells across all five notebooks, per Hide cells both in jupyterlab and myst rendering #335, and now carry numpydoc
docstrings. Three cells were split first, since a cell folds as a unit.

Why no coadds

Neither survey's coadds are reachable through SIA for the full release.

Roman WAS coadds exist in S3 but are not registered. IRSA is indexing the coadds, but
not on a timeline that affects this PR.
Rubin coadds were never generated. The preview did have one, which the original notebook used.

The notebooks use single exposures throughout and don't mention coadds, so nothing needs rewriting when the
Roman coadds are registered. Consequence for visualization notebook: every full-release Rubin product is a single visit, so the Rubin panel is genuinely shallow

CI cost

The firefly notebook was measured end-to-end, not estimated:

	data transferred	wall clock
before (coadd version)	4.9 GB	222 s
after	524 MB	229 s

Firefly: known issues, left for a separate PR

Firefly behaves inconsistently here. On some runs §5 and §7 work exactly as written — the Rubin image
displays and the region overlays appear — and on others the image does not display at all, on the same code
and inputs. I can't figure this out. This PR does not try to fix it. @jaladh-singhal please review sections 5 & 7 especially carefully, or open a new PR for any necessary changes.

firefly_client 3.4.0 also deprecated two things the notebook uses, so §5 and §7 print:

WARNING: show_fits() is deprecated. Use show_fits_image() instead.
WARNING: url is deprecated, use file_input parameter instead

Notes for reviewers

I tried to finagle so that the PR will show diffs and not new files for the notebooks which were edited, not sure I succeeded.
Also updated the top.yaml, but I can't figure out how to see what this looks like until I push.....so here goes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…tebook Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jaladh-singhal

This is mostly looking good @jkrick except a few things.

Because we no longer have coadds and full images are different than what the Firefly tutorial was written for originally, the narrative has become outdated in several places. Thanks for fixing the first half of it. For the second half, it will require some substantial edits and pruning to keep the narrative relevant and code working. Maybe I should do it in a separate PR based on this PR branch?

Re firefly issues with plotting, I think IRSAViewer is more finicky when running from Fornax due to the distance and/or load. I tried using NASA cloud firefly instead and don't see any issues except rubin 3 color image as I mentioned in comments below. Let's hope Fornax image gets updated by the time of workshop to use cloud firefly.

jaladh-singhal · 2026-08-25T00:23:37Z

+# Local working notes, not part of the tutorials
+CLAUDE.md
+OU_PR_REVIEW.md
+PR_DESCRIPTION.md


I think these can be removed

This could stay as it will help against accidental push of any of these files.

In that case only CLAUDE.md should stay - the other two md files seem specific to this PR and don't apply to our repo.

it doesn't need to apply to the repo, if e.g. Jessica creates these files for her claude then they should go in. If she doesn't use the files then I agree, it should not be in this file.
Bottom line, please add any files that doesn't belong in the repo but you have locally.

yes, I have them locally, and don't want them pushed to the repo, so I think this tays as is.

jaladh-singhal · 2026-08-26T02:35:34Z

+
+### Use Firefly's set_stretch method to change the stretch of the image display via Python
+
+The image has a lot of noise that obscures our high redshift sources of interest. You can use the Firefly GUI to change the stretch of the image display. We identify that squared stretch from -2 to 10 sigma highlights the colors of our sources better. You can also use the Firefly client's [`set_stretch`](https://caltech-ipac.github.io/firefly_client/api/firefly_client.FireflyClient.html#firefly_client.FireflyClient.set_stretch) to do this via Python. This is helpful for reproducibility and for scaling up to many images.


Maybe we should move it above and tweak the narrative re high redshift sources.

I'll leave this for now, but happy to switch the order of pan/zoom and stretch if that is your recommendation.

jaladh-singhal · 2026-08-26T02:56:42Z

+The saved gif loops without stopping, which makes a brief brightening hard to follow. Displaying the animation instead gives playback controls: pause it, step one epoch at a time with the arrows, drag the slider to any frame, and read the date in the corner as you go.
+
+```{code-cell} ipython3
+HTML(anim.to_jshtml(default_mode='once'))
+```


This renders nicely in fornax but I'm not sure how it will look in our static HTML build. For som reason, this tutorial wasn't executed so I can't tell: https://circle.scientific-python.dev/output/job/758ba48d-7674-427e-98ba-4d12fb345d12/artifacts/0/_build/html/openuniverse2024-roman-tds-supernova

The old timedomainsurvey notebook was on the ignore_circleci_testing list so when I renamed the tutorial, I kept it on that list. Agreed it's worth eyeballing on the site once this merges.

bsipocz

Thank you for the PR! I haven't noticed anything critical, most of my comments are about small things that are not blockers.
(I have not reviewed the firefly notebook, but looked through the diff for the other ones)

bsipocz · 2026-08-28T17:02:12Z

+# Local working notes, not part of the tutorials
+CLAUDE.md
+OU_PR_REVIEW.md
+PR_DESCRIPTION.md


This could stay as it will help against accidental push of any of these files.

bsipocz · 2026-08-28T17:21:00Z

 The OpenUniverse2024 data live on the cloud in a public AWS S3 bucket and can be accessed anonymously using `s3fs`. This section shows how to establish that connection, navigate the directory tree, and inspect the contents of a FITS image file.

-In the path below, `simple_model` refers to the simulated images with noise and realistic instrument effects, as opposed to `truth` images which are noise-free. The `full` simulation covers the complete survey footprint; a smaller `preview` subset is also available. See the [OpenUniverse2024 paper](https://arxiv.org/abs/2501.05632) for details on the differences. A `pointing` is a unique Roman observation visit — each pointing corresponds to one placement of the 18-detector focal plane on the sky, producing up to 18 individual FITS files (one per detector).
+In the path below, `simple_model` refers to the simulated images with noise and realistic instrument effects, as opposed to `truth` images which are noise-free. The `full` simulation covers the complete survey footprint. See the [OpenUniverse2024 paper](https://arxiv.org/abs/2501.05632) for details on how it was produced. A `pointing` is a unique Roman observation visit — each pointing corresponds to one placement of the 18-detector focal plane on the sky, producing up to 18 individual FITS files (one per detector).


Let's try and use DOIs so we get nice references. Also cut up the paragraph to one sentence per line.

Suggested change

In the path below, `simple_model` refers to the simulated images with noise and realistic instrument effects, as opposed to `truth` images which are noise-free. The `full` simulation covers the complete survey footprint. See the [OpenUniverse2024 paper](https://arxiv.org/abs/2501.05632) for details on how it was produced. A `pointing` is a unique Roman observation visit — each pointing corresponds to one placement of the 18-detector focal plane on the sky, producing up to 18 individual FITS files (one per detector).

In the path below, `simple_model` refers to the simulated images with noise and realistic instrument effects, as opposed to `truth` images which are noise-free.

The `full` simulation covers the complete survey footprint. See the [](https://doi.org/10.1093/mnras/staf1833) for details on how it was produced.

A `pointing` is a unique Roman observation visit — each pointing corresponds to one placement of the 18-detector focal plane on the sky, producing up to 18 individual FITS files (one per detector).

bsipocz · 2026-08-28T17:21:48Z

 ### 2.1 Inspect the SNANA Transient Catalog

-`inspect_parquet_columns()` reads only the Parquet metadata footer to print the row count and column names — no data is loaded into memory.
+`inspect_parquet_files()` reads only the Parquet metadata footer to print the row count and column names — no data is loaded into memory.


nice catch!

bsipocz · 2026-08-28T17:45:45Z

+from astroquery.ipac.irsa import Irsa
+from matplotlib import animation
+from scipy.ndimage import rotate
+from IPython.display import HTML


This is new, we'll need to declare ipython as a dependency, too.

added to both the requirements and the install line

bsipocz · 2026-08-28T17:52:34Z

+---
+def animate_stamps(stamps, savepath, no_whitespace=True,
+                   labels=[],labelxy=(0.05,0.95),
+                   **kwargs):


I think we could drop kwargs -- we don't use it when this function is called, and while the pattern can be useful in some cases it is also very prone to bugs and unexpected behaviour.

bsipocz · 2026-08-28T17:52:57Z

+    """
+    Make an animation of a sequence of image stamps.
+
+    :param stamps: Must be in chronological order.


at some point I think we should consistently switch to numpydoc formatting rather than this AI default.

FYI, this one was written by Lauren Alderoty from the OU team, not me. I try to remember to make all functions numpydocs style (obviously I am not perfect).

I did check that all functions in the OU notebooks are numpydocs style, and only found 2 which needed to be changed, so at least this small part of the repo is consistent.

bsipocz · 2026-08-28T17:56:35Z


+## OpenUniverse2024
+
+[OpenUniverse2024](https://arxiv.org/abs/2501.05632) simulates the overlapping imaging surveys to be carried out by the Nancy Grace Roman Space Telescope and the Vera C. Rubin Observatory, covering roughly 70 square degrees of matched optical and infrared sky.


Even if we don't change the reference in the notebooks for DOI, but here in the landing page I think we should switch to it, it will give a much nicer rendering than these direct external links.

Suggested change

[OpenUniverse2024](https://arxiv.org/abs/2501.05632) simulates the overlapping imaging surveys to be carried out by the Nancy Grace Roman Space Telescope and the Vera C. Rubin Observatory, covering roughly 70 square degrees of matched optical and infrared sky.

[OpenUniverse2024](https://doi.org/10.1093/mnras/staf1833) simulates the overlapping imaging surveys to be carried out by the Nancy Grace Roman Space Telescope and the Vera C. Rubin Observatory, covering roughly 70 square degrees of matched optical and infrared sky.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jkrick

Thanks for the comments @bsipocz and @jaladh-singhal . I think I got all of them, except most of the firefly comments which I prefer to leave for a separate PR since I was unable to get the testing working on Fornax for this.

The firefly deprecation PR might also want to touch Euclid_ERO.md and SEDs_in_Firefly.md, which call the same deprecated show_fits()

jkrick · 2026-08-31T13:50:48Z

+# Local working notes, not part of the tutorials
+CLAUDE.md
+OU_PR_REVIEW.md
+PR_DESCRIPTION.md


yes, I have them locally, and don't want them pushed to the repo, so I think this tays as is.

jkrick · 2026-08-31T17:19:24Z

+    """
+    Make an animation of a sequence of image stamps.
+
+    :param stamps: Must be in chronological order.


FYI, this one was written by Lauren Alderoty from the OU team, not me. I try to remember to make all functions numpydocs style (obviously I am not perfect).

I did check that all functions in the OU notebooks are numpydocs style, and only found 2 which needed to be changed, so at least this small part of the repo is consistent.

jkrick · 2026-08-31T20:29:07Z

+The saved gif loops without stopping, which makes a brief brightening hard to follow. Displaying the animation instead gives playback controls: pause it, step one epoch at a time with the arrows, drag the slider to any frame, and read the date in the corner as you go.
+
+```{code-cell} ipython3
+HTML(anim.to_jshtml(default_mode='once'))
+```


The old timedomainsurvey notebook was on the ignore_circleci_testing list so when I renamed the tutorial, I kept it on that list. Agreed it's worth eyeballing on the site once this merges.

jkrick · 2026-08-31T20:30:31Z


+## OpenUniverse2024
+
+[OpenUniverse2024](https://arxiv.org/abs/2501.05632) simulates the overlapping imaging surveys to be carried out by the Nancy Grace Roman Space Telescope and the Vera C. Rubin Observatory, covering roughly 70 square degrees of matched optical and infrared sky.


jkrick · 2026-08-31T20:34:00Z

 The OpenUniverse2024 data live on the cloud in a public AWS S3 bucket and can be accessed anonymously using `s3fs`. This section shows how to establish that connection, navigate the directory tree, and inspect the contents of a FITS image file.

-In the path below, `simple_model` refers to the simulated images with noise and realistic instrument effects, as opposed to `truth` images which are noise-free. The `full` simulation covers the complete survey footprint; a smaller `preview` subset is also available. See the [OpenUniverse2024 paper](https://arxiv.org/abs/2501.05632) for details on the differences. A `pointing` is a unique Roman observation visit — each pointing corresponds to one placement of the 18-detector focal plane on the sky, producing up to 18 individual FITS files (one per detector).
+In the path below, `simple_model` refers to the simulated images with noise and realistic instrument effects, as opposed to `truth` images which are noise-free. The `full` simulation covers the complete survey footprint. See the [OpenUniverse2024 paper](https://arxiv.org/abs/2501.05632) for details on how it was produced. A `pointing` is a unique Roman observation visit — each pointing corresponds to one placement of the 18-detector focal plane on the sky, producing up to 18 individual FITS files (one per detector).


jkrick · 2026-08-31T21:20:26Z

@jaladh-singhal I propose that we merge this PR and I will take you up on your offer of working on the firefly section of that one notebook in a separate PR. Please approve if this plan works for you.

jaladh-singhal

Thanks for addressing the comments @jkrick. Firefly notebook will need one more pass which I'll try to do this week as I find time.

Openuniverse tutorial updates ahead of Sep 15 webinar 4a263c1

jkrick self-assigned this Aug 15, 2026

jkrick added the content: openuniverse Content related issues/PRs for notebooks with OpenUniverse relevance label Aug 15, 2026

jkrick requested a review from troyraen August 15, 2026 01:28

troyraen requested review from bsipocz and jaladh-singhal and removed request for troyraen August 17, 2026 21:53

jkrick and others added 9 commits August 25, 2026 12:46

Rename OpenUniverse2024 tutorials to consistent filenames

8471594

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Update OpenUniverse2024 tutorials to use the full dataset

b5a54ef

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Add playback controls and visit-coverage guidance to the supernova no…

e92bd38

…tebook Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Fold helper function definitions into hidden cells

4ecbbcb

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Rework the firefly notebook to use single exposures

3db85bb

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Tighten the firefly notebook text and fix the high-redshift target

28b537d

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Restore the testing overrides and add reproject to the install line

b7a1401

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Ignore the local PR description note

7750d80

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Add s3fs back to the install line

a3175c6

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jaladh-singhal force-pushed the openuniverse-updates branch from 99ba4fa to a3175c6 Compare August 25, 2026 20:06

jaladh-singhal requested changes Aug 26, 2026

View reviewed changes

bsipocz reviewed Aug 28, 2026

View reviewed changes

bsipocz force-pushed the main branch from d972f95 to f208192 Compare August 28, 2026 22:27

jkrick and others added 3 commits August 31, 2026 13:14

Apply review feedback to the OpenUniverse2024 tutorials

2c2279e

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Use numpydoc docstrings throughout the OpenUniverse2024 tutorials

9f4a60c

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Rephrase the supernova epoch note and split the quickstart paragraph

544314d

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jkrick commented Aug 31, 2026

View reviewed changes

jaladh-singhal approved these changes Aug 31, 2026

View reviewed changes

jkrick merged commit 4a263c1 into Caltech-IPAC:main Sep 1, 2026
8 checks passed

github-actions Bot pushed a commit that referenced this pull request Sep 1, 2026

Merge pull request #370 from jkrick/openuniverse-updates

71f3187

Openuniverse tutorial updates ahead of Sep 15 webinar 4a263c1

This was referenced Sep 2, 2026

re-organize simulated data landing page #324

Closed

Update OU Firefly visualization tutorial after switching to full data release #377

Open


		### Use Firefly's set_stretch method to change the stretch of the image display via Python

		The image has a lot of noise that obscures our high redshift sources of interest. You can use the Firefly GUI to change the stretch of the image display. We identify that squared stretch from -2 to 10 sigma highlights the colors of our sources better. You can also use the Firefly client's [`set_stretch`](https://caltech-ipac.github.io/firefly_client/api/firefly_client.FireflyClient.html#firefly_client.FireflyClient.set_stretch) to do this via Python. This is helpful for reproducibility and for scaling up to many images.


		## OpenUniverse2024

		[OpenUniverse2024](https://arxiv.org/abs/2501.05632) simulates the overlapping imaging surveys to be carried out by the Nancy Grace Roman Space Telescope and the Vera C. Rubin Observatory, covering roughly 70 square degrees of matched optical and infrared sky.

	[OpenUniverse2024](https://arxiv.org/abs/2501.05632) simulates the overlapping imaging surveys to be carried out by the Nancy Grace Roman Space Telescope and the Vera C. Rubin Observatory, covering roughly 70 square degrees of matched optical and infrared sky.
	[OpenUniverse2024](https://doi.org/10.1093/mnras/staf1833) simulates the overlapping imaging surveys to be carried out by the Nancy Grace Roman Space Telescope and the Vera C. Rubin Observatory, covering roughly 70 square degrees of matched optical and infrared sky.

Uh oh!

Conversation

jkrick commented Aug 15, 2026

What changed

Why no coadds

CI cost

Firefly: known issues, left for a separate PR

Notes for reviewers

Uh oh!

jaladh-singhal left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bsipocz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jkrick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

jaladh-singhal left a comment •

edited

Loading