Allsky Raspberry Pi HQ Camera Dark Frame Temperature auto naming SOLVED !

[DSCustoms]

Taking dark frames with rpi HQ camera

To get the sensor temperature add --metadata /home/(your pi name)/allsky/config/overlay/extra/libcamera.txt --metadata-format txt  to the Extra Arguments setting in the WebUI. This will update the libcamera.txt file after every image with data about the image - a timestamp, the exposure used, the sensor temperature (if the camera supports it), and other data.

Do the following
cd allsky
cd scripts
sudo nano saveImage.sh

At the top, after /bin/bash paste the following code

# Set the path to the libcamera.txt file
LIBCAMERA_FILE="/home/(your user name or pi)/allsky/config/overlay/extra/libcamera.txt"

Then scroll down to line 124 and delete the following block of code

# If ${AS_TEMPERATURE_C} is set, use it as the sensor temperature,
# otherwise use the temperature in ${TEMPERATURE_FILE}.
# TODO: Currently nothing creates the TEMPERATURE_FILE.  Eventually RPi cameras will.
if [[ -z ${AS_TEMPERATURE_C} ]]; then
	TEMPERATURE_FILE="${ALLSKY_TMP}/temperature.txt"
	if [[ -s ${TEMPERATURE_FILE} ]]; then	# -s so we don't use an empty file
		AS_TEMPERATURE_C=$( < "${TEMPERATURE_FILE}" )
	fi
fi

Replace it with this code

# Modified Code
# Check if AS_TEMPERATURE_C is not set
if [[ -z ${AS_TEMPERATURE_C} ]]; then
        # Extract SensorTemperature value from libcamera.txt and assign it to AS_TEMPERATURE_C
        AS_TEMPERATURE_C=$(grep -oP 'SensorTemperature=(-?\d+)' "${LIBCAMERA_FILE}" | grep -oP '(-?\d+)')
fi

close ctrl-x
save y
enter
sudo reboot

Now the allsky software will save the temperature data from the rpi hq camera to the libcamera.txt file when you take photos, and create dark frames with the correct filenames!

Put the lens cover on, and put the whole enclosure in a dark bag, I use the one I handle photographic film in.

Set “take dark frames” to yes. Save and restart.

I put the whole thing in the freezer for an hour, then start it back up while taking dark frames and just log every photo. Do this a few times and you’ll get a nice range of temps.

When you are bored of this, turn off “take dark frames” and turn on “use dark frames”

Took a bunch of effort to figure out as I barely code :)

[DSCustoms]

Please let me know if it works for you, or any improvements. I just set up my allsky thanks to this community and noticed this still wasn't fixed.

[EricClaeys]

@DSCustoms, you really don't want to be modifying Allsky code like saveImage.sh since your changes will be overwritten whenever you upgrade, and when users make changes to our code it makes it difficult for us to support.

I suggest adding a "Allsky Script" module prior to the "Overlays data on the image" module, where the script would do your "grep" and write the output to the TEMPERATURE_FILE (${ALLSKY_TMP}/temperature.txt").

If that works we would likely make your script an official one so others could use it. Of course once I get off my butt and update Allsky to handle temperature for RPi cameras that support it, the change won't be needed.

[DSCustoms]

Thanks I’m new to contributing on these. I’ve been doing small code projects on things for decades and noticed this little thing that didn’t work. I’m sure you will all get around to it, I just got bored doing them manually and wanted a quick fix 😂

[MadMacMad]

many thx!
like it ... and yes if possible plz make it a extra script so that everybody can use it like Eric said...
thank you :-)

[DSCustoms]

Ok, so I'm making a file in /allsky/ called rpi_hq_temp.sh
File contents are

#!/bin/bash

USER_HOME=${HOME}

if [ $EUID -eq 0 ]; then
    # Running with sudo, get the home directory of the invoking user
    USER_HOME=$(getent passwd "$SUDO_USER" | cut -d: -f6)
fi

LIBCAMERA_FILE="${USER_HOME}/allsky/config/overlay/extra/libcamera.txt"
TEMPERATURE_FILE="${USER_HOME}/allsky/tmp/temperature.txt"

# Extract SensorTemperature value from libcamera.txt and always write it to TEMPERATURE_FILE
grep -oP 'SensorTemperature=(-?\d+)' "${LIBCAMERA_FILE}" | grep -oP '(-?\d+)' > "${TEMPERATURE_FILE}"

chatgpt had to help me with this a little. ;) Can a real programmer fix it?

then run chmod +x rpi_hq_temp.sh

adding the allsky_script module before "overlays data on an image" on both day and night capture flows in the module manager, set the file location as /home/(my dir)/allsky/rpi_hq_temp.sh

Slap on the lens cap, turn on
aaaaaaannnnd....
it still isn't working.

@EricClaeys can you confirm what module flow the dark frames are taken with? I put the script in day and night capture. I know the script works, I can run it manually and it'll create the temperature file, which then is picked up by saveImage.sh and saved into the filename of the dark. But I can't get the script module to run my code and the debug doesn't work. What am I missing?

[EricClaeys]

@Alex-developer,
Can the "script" module call shell scripts?

@DSCustoms, thanks for doing this. I will make some changes depending on what Alex says.
Darks can run during the day or night.

[Alex-developer]

@EricClaeys yes it can run any executable

[DSCustoms]

The docs said it can run any code. Am I just having an issue with where it finds the code? Did I need to locate the program in a better location?

[DSCustoms]

Ok, so I didn't hear back and didn't look at this for a few days. I was just trying to install some other code and tried something else, I put the allsky script module in every flow, now it all works fine. I guess Darks only run on the "Periodic Jobs" flow. Either way, instructions are now completely running. Thanks for the help!

[EricClaeys]

@DSCustoms, FYI, I got involved with Allsky because I noticed a lot of little things that "didn't work", just like you!

[EricClaeys]

@DSCustoms,

The scripts should never run as sudo - it'll hose up the file permissions. Below is a modified version of your script.

You may want to put your script somewhere other than ~/allsky, because it'll get remove whenever you upgrade. I put my scripts in ~ or ~/myscripts/.

What did you mean by:

I guess Darks only run on the "Periodic Jobs" flow.

You can take darks during daytime or nighttime but darks are only subtracted at nighttime. That's all done during the initial save of the image to allsky/tmp.
If your scripts runs in the Periodic flow it'll create the temperature file which will be used on the next image. It will always be 1 or 2 pictures behind, but that shouldn't matter since the temperature doesn't change drastically between pictures.

#!/bin/bash

source "${ALLSKY_HOME}/variables.sh"	|| exit 1

LIBCAMERA_FILE="${ALLSKY_EXTRA}/libcamera.txt"
TEMPERATURE_FILE="${ALLSKY_TMP}/temperature.txt"

# Extract SensorTemperature value from libcamera.txt and always write it to TEMPERATURE_FILE
sed --quiet -e '/^SensorTemperature=/ s/SensorTemperature=//p' "${LIBCAMERA_FILE}" > "${TEMPERATURE_FILE}"
if [[ $? -ne 0 ]]; then
	echo "ERROR: unable to create '${TEMPERATURE_FILE}'" >&2
	exit 1
fi
exit 0

[DSCustoms]

I can move the scripts. Working on some more to add temperatures from a sensor and graph them for now. But what I meant by the module flows, in the module manager allsky would never run my script while the “allsky script” module was used in the daytime or night time capture flows. ONLY in the periodic flow section would the script be run to put the new temperature into the txt file to be appended to the name of the dark file. If taking darks is enabled and the script module is in place where you originally recommended it before the overlay module runs in daytime or night time, it did not run.

[EricClaeys]

Not sure why the script wouldn't run in daytime or nighttime flows, but doesn't really matter I guess since you want it in the periodic flows anyway.

[DSCustoms]

Sorry, ignore that maybe I’m just confused about how it works. The script has to be in the periodic jobs flow for it to run during dark frame capture. That’s all.

[aaronwmorris]

For libcamera cameras, it should be noted that you should only take dark frames when two conditions are true:

1. Using PNG for output. You need a lossless source format and a lossless frame to properly subtract. DNG would be better, but I do not believe allsky supports that. If you use a JPEG as a dark frame, you will subtract compression artifacts from your images.
2. Auto-White Balance is disabled. If you have AWB enabled, the color values of noise will be adjusted before the image is generated, which means the wrong data will be subtracted.

[nacho3g]

Is there a way to choose PNG by editing a script?

[DSCustoms]

The settings mention png files output but I’m not quite sure how to change to that or if it’s camera dependent.

[EricClaeys]

When taking dark frames Allsky disables Auto-white balance.

To use png change the Filename in the WebUI to image.png.
Note that I haven't tested using a .png dark and a .jpg light so am not sure if it will work. It will in a future release when Allsky supports darks on RPi cameras.
If it doesn't work, set the quality to 100 for jpg files. I agree with Aaron but .jpg files appear to be better than nothing, at least on ZWO cameras they are.

[aaronwmorris]

On a 4K resolution camera (HQ camera), using a JPEG as a source for a dark frame will probably not be noticeable unless you are pixel-peeping. On a 1K resolution camera, the JPEG compression will probably start to become obvious.

The other thing that could help is adjusting chroma sub-sampling for JPEG. I have not played with this option, YMMV.

https://vovkos.github.io/doxyrest-showcase/opencv/sphinxdoc/enum_cv_ImwriteFlags.html (IMWRITE_JPEG_CHROMA_QUALITY)

https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html#jpeg (subsampling)

[troublestarter]

This "new code" would be integrated in a futur release ?

[EricClaeys]

@troublestarter, the next Allsky release will not include changes to darks. Hopefully the release after. I would like to default to .png darks, and allow separate darks per camera type/model, and include a script to take darks using different settings (gain, exposure, etc).

[troublestarter]

good news for this. Thanks you

[ssagerian]

Im confused about this whole thread, I cant find any documentation on the RPi HQ camera supporting internal temperature reading. I added the extra arguments to the libcamera-still and see that its writing a file but the temperature value in the metadata file is always zero.
pi@homesky:~ $ cat allsky/config/overlay/extra/libcamera.txt
ExposureTime=114
SensorTimestamp=42210455074000
ScalerCrop=(0, 0)/4056x3040
SensorBlackLevels=[ 4096, 4096, 4096, 4096 ]
ColourTemperature=6496
DigitalGain=1.000000
ColourGains=[ 3.691746, 1.497074 ]
AeLocked=true
SensorTemperature=0.000000
Lux=19522.771484
FrameDuration=100000
ColourCorrectionMatrix=[ 2.006445, -0.832742, -0.173694, -0.269630, 1.811624, -0.542002, -0.110392, -0.497065, 1.607457 ]
AnalogueGain=1.000000
FocusFoM=9710

Can someone set me straight? thanks

[aaronwmorris]

There were some bugs in the linux kernel last year that prevented getting the temperature from the HQ camera (IMX477) and Camera Module 3 (IMX708). Make sure you update your kernel packages for the Raspi (and reboot).

raspberrypi/rpicam-apps#523

[ssagerian]

Yep, that worked, thanks!

[gingerbreadassassin]

For those coming to this thread looking for a more turn-key how-to, here's how I got temperatures working for dark frames (Pi 4 + HQ camera + Allsky Version: v2023.05.01_04).

1. Run sudo rpi-update to get the latest kernel modules (fixes temperature always reported as zero)
   a. There are some scary warnings when running this. If someone knows a way to install the specific kernel module that fixes the problem, please do share. I had no issues, though.
   b. Follow up with sudo reboot
2. Save images as "png" by setting Allsky Settings > Both daytime and nighttime settings > Filename to image.png
   a. Be sure to set Allsky Settings > Both daytime and nighttime settings > Quality / Compression accordingly (0-9 for png)
3. Save the image metadata by adding --metadata /home/pi/allsky/config/overlay/extra/libcamera.json to Allsky Settings > Both daytime and nighttime settings > Extra Arguments
4. Create a shell script /home/pi/custom-allsky-scripts/get_temp.sh (don't forget to mark executable via chmod +x) with contents:

#!/bin/bash

source "${ALLSKY_HOME}/variables.sh"    || exit 1

LIBCAMERA_FILE="${ALLSKY_EXTRA}/libcamera.json"
TEMPERATURE_FILE="${ALLSKY_TMP}/temperature.txt"

# Extract SensorTemperature value from libcamera.txt and always write it to TEMPERATURE_FILE
/usr/bin/jq -r .SensorTemperature "${LIBCAMERA_FILE}" > "${TEMPERATURE_FILE}"
if [[ $? -ne 0 ]]; then
        echo "ERROR: unable to create '${TEMPERATURE_FILE}'" >&2
        exit 1
fi
exit 0

5. In Module Manager > Daytime Capture, drag the AllSKY Script module to the right, positioned so:
   
6. Edit the AllSKY Script module's Settings and fill in the File Location so that it reads /home/pi/custom-allsky-scripts/get_temp.sh
7. Repeat 5 & 6 for Module Manager > Nightime Capture
8. Restart Allsky

Let your darks run (reference the documentation if unsure how) and you should get properly named darks in the folder:

pi@allsky:~ $ ls allsky/darks/
74.png  75.png

I'll come back to edit this later with whether or not temperatures are properly reported in the overlays as well.

EDIT: Temperature is working in the overlay

Followed the "v2023.05.01_02 - Point Release # 2" instructions in the change log to add the variable.

[gingerbreadassassin]

Oddly enough, I never ran into that issue. Quality/Compression were still set to 100 😅. Will add that to the write-up.

Are you also running on a Pi 4?

[EricClaeys]

@cdoebert,
We're you using Quality of 100 with a .png file? If so, what exactly was taking 85 seconds to save? The initial save to allsky/tmp/image.png, or the final save by the Overlay module?

png files don't have a quality - they are always at the highest quality but have a compression for 0 to 9 and I am pretty sure the higher the compression the longer the save time.

[SkyDragon2023]

I've tried to follow the instructions, but after adding the Allsky Script module, editing the settings and leaving it to run I get a message in the debug window stating ran but returns a false result: "allsky_script.py 0.01 false"
If I run the script from the command line via putty it returns this error:
./get_temp.sh
: No such file or directoryile /home/pi/allsky/config/overlay/extra/libcamera.json
./get_temp.sh: line 11: syntax error: unexpected end of file
The libcamera.json file is present in the required folder and its updating.
I made the script file in notepad++ on windows saved it as a .sh file and copied it over to the folder on my pi, chmoded as stated above but it doesn't work.

[EricClaeys]

@SkyDragon2023,
It could be one of two things.

Windows and Linux mark the end of a line in a text file differently. Windows uses a carriage return AND a line feed, and Linux uses just a line feed. Since you made the file using a Windows program it probably has the Windows format, and when Linux tries to read the file it sees a line feed at the end of each line and treats it like a "normal" character, so on the first line it looks for /bin/bashX where X is a line feed, and it can't find that program. Then the lines after that fail as well.
Notepad++ probably has an option to save the file in Linux format.

The other possibility is you have a syntax error. unexpected end of file usually means bash was looking for something, often a fi or a closing double quote and got to the end of the file before bash found it.

If you cannot figure it out attach, DO NOT copy/paste the file.

[DSCustoms]

If I’m copying code these days I just paste it into chat gpt and say format this correctly for my (xxx system) . I tend to always use the wrong “‘ or some other character or indentation that isn’t quite right, but gpt will spit out a working script pretty much 100% of the time. For people who aren’t actual programmers (like me) this is probably the best tool out there.

[EricClaeys]

@gingerbreadassassin, Connor, thanks for doing this. Please note that only a few of the RPi cameras support temperature.

[gingerbreadassassin]

Yep. I assumed that if you've made it this far trying to get temperature to work with the RPi camera, you must have come across that particular caveat.

[troublestarter]

i'm waiting for this ! 🥇

[ThomasW69]

@gingerbreadassassin

I have created my darkframes with your script and they are also in the corresponding directory. So far so good. However, it seems to me that Allsky then always uses the same dark frame when recording, regardless of the temperature. In my opinion, Allsky only uses the internal camera temperature during dark frame correction, which can only be read out via the script for RPi cameras, but not the generated "${ALLSKY_TMP}/temperature.txt". How do I get the evaluation via the "${ALLSKY_TMP}/temperature.txt" into the processing of the images?

[gingerbreadassassin]

Allsky then always uses the same dark frame when recording

How did you determine this? Is it in the logs somewhere?

[ThomasW69]

it is because over the night there are temperature changes of about 4 degrees Celsius but in the video i have some dark pixels. This are hot pixels that only appear in the darkframes over 30 degree Celsius.
https://www.sternwarte-kirchheim.de/Allsky/videos/allsky-20240503.mp4

[gingerbreadassassin]

Ah, I think I see. I circled a few of them:

So Allsky is subtracting hot pixels only present in 30C darks even from ~15C subframes.

For what it's worth:

https://htmlpreview.github.io/?https://raw.githubusercontent.com/AllskyTeam/allsky/master/html/documentation/explanations/darkFrames.html

https://htmlpreview.github.io/?https://raw.githubusercontent.com/AllskyTeam/allsky/master/html/documentation/knownIssues.html

I'm not sure on the details of the issue here; whether it's because of the need for a workaround to grab the temperature, and that's what's stopping subtraction from working, or some other complication. Perhaps a discussion on a workaround (or a github issue) already exists and just isn't linked to in the documentation. But I actually don't know where in the processing pipeline subtraction takes place; I haven't dug that deep into the code. If I have some spare time today, I'll try to take a look.

[EricClaeys]

Set Debug Level to 4 and restart Allsky. /var/log/allsky.log will tell you what dark frame it's using for every image. If it's wrong, let me know.

Dark subtraction is done in darkSubtract.sh which is called from saveImage.sh.

[EricClaeys]

@ThomasW69,
Can you explain your sentence:

Allsky only uses the internal camera temperature during dark frame correction, which can only be read out via the script for RPi cameras, but not the generated "${ALLSKY_TMP}/temperature.txt".

Are you saying that for RPi cameras the temperature.txt file is not being read? If so, then it's a bug and no dark subtraction will occur. The temperature.txt file is looked at in saveImage.sh.