Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
a178fdc
bump all product versions to 2
nicHoch Dec 19, 2023
3a06055
add ecc context manager
nicHoch Jul 22, 2025
3848748
add elut manager
nicHoch Jul 22, 2025
067651d
L2 calibration spectra and L2 sci-spectra
nicHoch Jul 22, 2025
d4b0ea5
update ecc conf
nicHoch Sep 17, 2025
641354d
prep fits file for post ecc fit
nicHoch Sep 23, 2025
9ec1392
integration of ecc post fit step with own conf
nicHoch Sep 30, 2025
8f876ad
alternate ebin edges for direct usage with data
nicHoch Oct 14, 2025
c45320f
add alternative e bin edges definition based on/not on applied ELUT
nicHoch Oct 23, 2025
941b581
fix for multiple spectra per day and more logging
nicHoch Nov 14, 2025
b1a7447
fix typos
nicHoch Dec 4, 2025
a23eb33
create a own CAL product "level"
nicHoch Jan 20, 2026
958b970
fix soop startup and add check for elut same at the end
nicHoch Jan 29, 2026
8f0b116
add CAL to publishing step
nicHoch Feb 10, 2026
4355806
cleanup
nicHoch Feb 10, 2026
4e7dd47
add missing dependencies
nicHoch Feb 10, 2026
1fb8bdf
undo SOOP API deactivation
nicHoch Feb 12, 2026
f924fa8
fixes after review
nicHoch Feb 12, 2026
7e966a5
remove obsolete file
nicHoch Feb 12, 2026
7c59ac0
fixes after review
nicHoch Feb 12, 2026
2e3480b
Update stixcore/calibration/ecc_post_fit.py
nicHoch Feb 12, 2026
d35d950
final fixes
nicHoch Feb 23, 2026
eeb3479
Merge branch 'L2_spec' of https://github.com/nicHoch/STIXCore into L2…
nicHoch Feb 23, 2026
c10c49d
go on with next spectra in same file in case of bad format
nicHoch Feb 27, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/end2end_run.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ jobs:
python -m pip install -e .[dev]
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: show environment
run: python -m pip freeze
run: python -m pip freeze
- name: end2end test with pytest
run: |
pytest -p no:warnings --doctest-rst -m "end2end" --pyargs stixcore --remote-data=any
31 changes: 0 additions & 31 deletions docs/pipelineconfiguration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -175,37 +175,6 @@ without provided arguments the default values from ``stixcore.ini`` are used
0 5 * * * cd /home/stixcore/STIXCore/ && /home/stixcore/STIXCore/venv/bin/python /home/stixcore/STIXCore/stixcore/processing/publish.py --update_rid_lut


Run the pipeline monitor
************************

The event based pipeline (observing incoming telemetry files) gets stuck from time to time. There is a process observing the number of open to process files. If the number of open files is constantly increasing over a longer period a notification mail is send out:

.. code-block::

# run pipeline monitor task to check for pipeline not stuck
0 */3 * * * cd /home/stixcore/STIXCore/ && /home/stixcore/STIXCore/venv/bin/python /home/stixcore/STIXCore/stixcore/processing/pipeline_monitor.py -s /home/stixcore/monitor_status.json


In case of a pipeline stuck restart the event based processing pipeline.

.. code-block::

# stop the system.d process
sudo systemctl stop stix-pipeline.service

# wait 20sec so that all open sockets also gets closed
# start the process again

sudo systemctl start stix-pipeline.service

In order to process all tm files that have not been processed so fare the config parameter start_with_unprocessed should be set to true:

.. code-block::

[Pipeline]
start_with_unprocessed = True


Run the 'daily' pipeline
************************

Expand Down
Binary file added processing_history.sqlite
Binary file not shown.
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ dependencies = [
"numpy==2.3.4",
"sunpy~=6.0.0",
"watchdog~=6.0",
"lmfit==1.3.4",
"stixdcpy==3.0",
"stixpy>=0.2.0rc1,<0.3"
]
Expand Down
229 changes: 229 additions & 0 deletions stixcore/calibration/ecc_post_fit.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,229 @@
"""
Author: O. Limousin, CEA
Date: Oct 23, 2024
This script to:

#%%%%%%%%%%%%%% FOR fit of ECC_ULTRA_FINE %%%%%%%%%%%%%%
../00_CALIBRATION_MONITORING_ULTRA_FINE/02_MONITOR_ECC_SPECTRA ==> XX_erg.fits

../00_CALIBRATION_MONITORING_ULTRA_FINE/01_MONITOR_ECC_PARA/ ==> ECC_para_XXXX.fits

+ read the log (.xlsx file)
+ open the ECC calibrated spectra XX_erg.fits
+ fit 31 and 81 keV lines (Fit_Ba_robust) (fit right hand side, and baseline)
+ register the results in a Pandas Dataframe
+ date, Run Number, Fit Goodness flags,DET, PIX,
P31, err_P31, dE31, err_dE31, H31
P81, err_P81, dE81, err_dE81, H81
+ compute gain/offset corrections
+ fill the dataframe with new values
+ include errors
+ include a Flag (Flag31 and Flag81) which is True if fit was OK
if was not OK, the ECC value in NOT corrected
+ store the results in pkl
+ generate update ECC_para files to recalibrate uncalibrated files

"""

import numpy as np
import pandas as pd
from lmfit import Model

from astropy.io import fits

from stixcore.util.logging import get_logger

__ALL__ = ["ecc_post_fit_on_fits", "ecc_post_fit"]
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python convention is to use __all__ (lowercase), not __ALL__ (uppercase). This won't cause a runtime error but deviates from PEP 8 and won't be recognized by tools that use the standard __all__ dunder.

Suggested change
__ALL__ = ["ecc_post_fit_on_fits", "ecc_post_fit"]
__all__ = ["ecc_post_fit_on_fits", "ecc_post_fit"]

Copilot uses AI. Check for mistakes.

logger = get_logger(__name__)


def open_fits_tables(fits_path):
# Get the data from .fits
data = []
with fits.open(fits_path, memmap=False) as hdul:
header = [hdul[0].header]
for i in range(1, len(hdul)):
header.append(hdul[i].header)
data.append(hdul[i].data)

return header, data


def Read_fits_STIX_One_Pixel(data, PIX, DETECTOR_ID, Nbin=2024, NRebin=1, NSigma=1):
Pix = PIX
Pix = Pix + (DETECTOR_ID - 1) * 12
erg_c = data.ERG_center # data.field(0)
obs = data.field(3 + Pix)
# determine number of bins to use, taking into account any requested rebinning
nbin = int(Nbin / NRebin)
erg_c = erg_c[:nbin]
obs = obs[:nbin]

yerr = NSigma * np.sqrt(obs)
return erg_c, obs, yerr


def line(x, slope, intercept):
"""a line"""
return slope * x + intercept


def poly(x, degree, slope, intercept):
"""a line"""
return degree * x * x + slope * x + intercept


def gaussian(x, amp, cen, wid):
# """1-d gaussian: gaussian(x, amp, cen, wid)"""
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commented-out docstring for the gaussian function. Either uncomment it or remove it to keep the code clean.

Suggested change
# """1-d gaussian: gaussian(x, amp, cen, wid)"""
"""1-d gaussian: gaussian(x, amp, cen, wid)"""

Copilot uses AI. Check for mistakes.
return amp * np.exp(-((x - cen) ** 2) / (2 * wid**2))


def Fit_Ba_Lines_Robust(erg_c, obs):
"""
OL, oct 22, 2024
Robust fit procedure to adjust peak position post ECC
The idea is the exact same as previous function but to return
+ 0's in case the fit fails
+ flag to say if the fit was successful or not
"""
# Select Energy range for 81 keV Ba-133 line
pipo = ((erg_c > 80.5) & (erg_c < 90.0)).nonzero()
y = obs[pipo]
x = erg_c[pipo]
x = np.array(x, dtype="float64")
y = np.array(y, dtype="float64")
mod = Model(gaussian) + Model(line)
pars = mod.make_params(amp=10, cen=81, wid=0.5, slope=0, intercept=0)
result = mod.fit(y, pars, x=x)

if (
(result.params["wid"].stderr is not None)
& (result.params["cen"].stderr is not None)
& (np.abs(result.best_values["cen"] - 81.0) < 1.0)
):
dE81 = result.best_values["wid"] * 2.35
err_dE81 = result.params["wid"].stderr * 2.35
P81 = result.best_values["cen"]
err_P81 = result.params["cen"].stderr
H81 = result.best_values["amp"]
Goodness_Flag_81 = True
else:
dE81, err_dE81, P81, err_P81, H81 = 0, 0, 0, 0, 0
Goodness_Flag_81 = False

# Select Energy range for 81 keV Ba-133 line
# THE FOLLOWING SECTION IS QUIET ROBUST BUT MIGHT OVERESTIMATE THE ENERGY RESOLUTION AT 32 keV
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in comment: "QUIET" should be "QUITE".

Suggested change
# THE FOLLOWING SECTION IS QUIET ROBUST BUT MIGHT OVERESTIMATE THE ENERGY RESOLUTION AT 32 keV
# THE FOLLOWING SECTION IS QUITE ROBUST BUT MIGHT OVERESTIMATE THE ENERGY RESOLUTION AT 32 keV

Copilot uses AI. Check for mistakes.
# THIS ALLOWS TO FORCE THE BASE LINE TO ADJUST in 40-45 keV range while a simple Gaussian
# is used to fit the right hand side of the 32 keV Line
pipo = (((erg_c > 30.2) & (erg_c < 33.0)) | ((erg_c > 40) & (erg_c < 45))).nonzero()
y = obs[pipo]
x = erg_c[pipo]
x = np.array(x, dtype="float64")
y = np.array(y, dtype="float64")

mod = Model(gaussian, prefix="g1_") + Model(poly)
pars = mod.make_params(g1_amp=10, g1_cen=30.6, g1_wid=0.4, degree=0.0, slope=0, intercept=0.0)

result = mod.fit(y, pars, x=x)

if (
(result.params["g1_wid"].stderr is not None)
& (result.params["g1_cen"].stderr is not None)
& ((np.abs(result.best_values["g1_cen"] - 31.0) < 1.0) & (Goodness_Flag_81))
):
dE31 = result.best_values["g1_wid"] * 2.35
err_dE31 = result.params["g1_wid"].stderr * 2.35
P31 = result.best_values["g1_cen"]
err_P31 = result.params["g1_cen"].stderr
H31 = result.best_values["g1_amp"]
Goodness_Flag_31 = True
else:
dE31, err_dE31, P31, err_P31, H31 = 0, 0, 0, 0, 0
Goodness_Flag_31 = False

return P31, P81, dE31, dE81, err_P31, err_P81, err_dE31, err_dE81, H31, H81, Goodness_Flag_31, Goodness_Flag_81


def ecc_post_fit_on_fits(erg_file, para_file, livetime):
data_erg = open_fits_tables(erg_file)[1][0] # only need data and only the first table which contains the spectra
data_para = open_fits_tables(para_file)[1][
0
] # only need data and only the first table which contains the ECC parameters

gain = np.float32(data_para.gain)
off = np.float32(data_para.off)
goc = np.float32(data_para.goc)

return ecc_post_fit(data_erg, gain, off, goc, livetime)


def ecc_post_fit(data_erg, gain, off, goc, livetime):
DETs = np.arange(32) + 1
LARGEs = [0, 1, 2, 3, 4, 5, 6, 7]
SMALLs = [8, 9, 10, 11]
PIXELs = LARGEs + SMALLs

accumulator = []

# Proceed to fit individually each pixel spectrum in the list of files
for DET in DETs:
for PIX in PIXELs: # or in LARGEs, SMALLs
erg_c, obs, yerr = Read_fits_STIX_One_Pixel(data_erg, PIX=PIX, DETECTOR_ID=DET)

P31, P81, dE31, dE81, err_P31, err_P81, err_dE31, err_dE81, H31, H81, Goodness_Flag31, Goodness_Flag81 = (
Fit_Ba_Lines_Robust(erg_c, obs / livetime)
)

dict = {
"DET": [DET],
"PIX": [PIX],
"P31": [P31],
"err_P31": [err_P31],
"dE31": [dE31],
"err_dE31": [err_dE31],
"Flag31": [Goodness_Flag31],
"P81": [P81],
"err_P81": [err_P81],
"dE81": [dE81],
"err_dE81": [err_dE81],
"Flag81": [Goodness_Flag81],
}
Comment on lines +178 to +191
Copy link

Copilot AI Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shadowing the built-in dict with a variable name will cause issues if the built-in is needed elsewhere in the function. While it works here, it's a Python anti-pattern. Rename this variable to something like pixel_data or result_dict.

Copilot uses AI. Check for mistakes.
logger.debug(dict)

# NB: this is faster to append a list of dict and create DataFrame at the end
# than concatenating the DataFrame row by row, this slows down dramatically progressively
accumulator.append(pd.DataFrame(dict))

df = pd.concat(accumulator)
df = df.reset_index(drop=True)

# 3- gain and offset correction factors of ECC pre-calibrated data
G_prime = (df["P81"] - df["P31"]) / (80.9979 - (30.6254 * 33.8 + 30.9731 * 62.4) / (62.4 + 33.8))
O_prime = df["P31"] - G_prime * (30.6254 * 33.8 + 30.9731 * 62.4) / (62.4 + 33.8)

# 4- add correction factors to the DataFrame
df["Gain_Prime"] = G_prime
df["Offset_Prime"] = O_prime

# check, Run number and pixel number prior to assign Gain and Offset for further correction
df["Gain_ECC"] = gain
df["Offset_ECC"] = off
df["goc"] = goc

# 7 - Compute corrected Gain and Offset and fill df
# 7.2 - Now assign the corrected Gain and Offset values
# except when ECC works better
df["Gain_Cor"] = 0.0
df["Offset_Cor"] = 0.0

# apply correction to gain and offset when fit is ok
idx = df["Flag31"] & df["Flag81"]
df.loc[idx, "Gain_Cor"] = df["Gain_ECC"][idx] * df["Gain_Prime"][idx]
df.loc[idx, "Offset_Cor"] = df["Gain_ECC"][idx] * df["Offset_Prime"][idx] + df["Offset_ECC"][idx]
# otherwise keep uncorrected ECC Values
idx = ~idx
df.loc[idx, "Gain_Cor"] = df["Gain_ECC"][idx]
df.loc[idx, "Offset_Cor"] = df["Offset_ECC"][idx]

return df, idx
Loading