Added in a flux_err mask to remove NaNa and fix Issue 1114

[rebekah9969]

I have added in a NaN mask for flux_err and combined with the flux mask such that the PLD corrector can function.

[rebekah9969]

fixes #1114

[tylerapritchard]

Notes from our Lightkurve Party Discussion:

(CH: Ignore below:)

• What happens with HLSPs where the flux_err column is NaN because there are no errors (TICA/K2SFF)?
• What happens when these are masked values and/or astropy quantities?
• Does this account for masked arrays not just values? Masks themselves have values which are masked
• This needs a test if it doesn't already have one. This could be putting something into regression corrector, masking it, then doing it again. If this exists then we're good.

[rebekah9969]

1. To use PLDCorrector a TPF object is required. This object must have flux_err values.

The only HLSP products that would qualify for PLD correction would be TICA cut outs and TESS-SPOC products.

TESS-SPOC products work fine with the PR code.

TICA products do not work because they have zeros for the flux_err values. The failure occurs at this point and not because of the added PR code. The error produced is
ValueError: Input light curve contains flux uncertainties smaller than or equal to zero. Please remove these (e.g. using lc = lc[lc.flux_err > 0])

2. The TPF object data and LC data are stored as astropy arrays with units of e/s and as such the code works with astropy quantities.

3. I have tried using astropy.utils.masked function to mask the tpf and as such obtain values which have --- as the quantity.

Following code i have used is

import numpy as np

from lightkurve.correctors import PLDCorrector
from lightkurve import TessTargetPixelFile as TPF

from astropy.utils.masked import Masked

lcf = "https://archive.stsci.edu/missions/tess/tid/s0026//0000/0001/5832/4245/tess2020160202036-s0026-0000000158324245-0188-s_tp.fits"

tpf = TPF(lcf)
lcs = tpf.to_lightcurve(aperture_mask='threshold')

mask = np.isnan(lcs.flux)
lcsm = Masked(lcs, mask=mask)
tpfm = Masked(tpf, mask=mask)

However, while this works on the lc it does not work on the tpf. For the tpf it takes a very long time to process and a warning is returned.
RuntimeWarning: invalid value encountered in double_scalars percent_masked = 100.0 * n_cadences_masked / n_cadences

4. There are several tests in test_pldcorrector.py
   They look at Kepler and TESS targets, test the aperture mask, and test tpf with zero flux cadences. I am unsure what additional test would be required.

[christinahedges]

• Make a TPF with all nan errors, and check that this fails gracefully
• Make a TPF with partial nan errors, and check that passes
• Capture both of the above as tests

[rebekah9969]

Test to check that a TPF with all NaN values for flux_err will fail the PR code.

import lightkurve as lk
import numpy as np

import matplotlib.pyplot as plt
plt.rcParams.update({'font.size': 15})
%matplotlib inline

from lightkurve.correctors import PLDCorrector
from lightkurve import TessTargetPixelFile as TPF

from astropy.io import fits

s=lk.search_targetpixelfile("KT Eri")

tpf = s[0].download()
fname = tpf.path

a = np.empty((112388,11,11))
a[:] = np.nan 

with fits.open(fname, mode='update') as hdulist:
    hdulist[1].data['FLUX_ERR'] = a
    
    print(hdulist[1].data['FLUX_ERR'].size)
    hdulist.close()

tpf = lk.read(fname)
pld = PLDCorrector(tpf)
corrected_lc = pld.correct()

This results in the following error and as such the corrector fails because the NaN mask effectively wipes out the entire data set.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[23], line 1
----> 1 corrected_lc = pld.correct()

File ~/miniforge3/lib/python3.10/site-packages/astropy/utils/decorators.py:546, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    543             msg += f'\n        Use {alternative} instead.'
    544         warnings.warn(msg, warning_type, stacklevel=2)
--> 546 return function(*args, **kwargs)

File ~/miniforge3/lib/python3.10/site-packages/astropy/utils/decorators.py:546, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    543             msg += f'\n        Use {alternative} instead.'
    544         warnings.warn(msg, warning_type, stacklevel=2)
--> 546 return function(*args, **kwargs)

    [... skipping similar frames: deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper at line 546 (1 times)]

File ~/miniforge3/lib/python3.10/site-packages/astropy/utils/decorators.py:546, in deprecated_renamed_argument.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    543             msg += f'\n        Use {alternative} instead.'
    544         warnings.warn(msg, warning_type, stacklevel=2)
--> 546 return function(*args, **kwargs)

File ~/miniforge3/lib/python3.10/site-packages/lightkurve/correctors/pldcorrector.py:405, in PLDCorrector.correct(self, pld_order, pca_components, pld_aperture_mask, background_aperture_mask, spline_n_knots, spline_degree, normalize_background_pixels, restore_trend, sparse, cadence_mask, sigma, niters, propagate_errors, use_gp, gp_timescale, aperture_mask)
    402     else:
    403         normalize_background_pixels = False
--> 405 dm = self.create_design_matrix(
    406     pld_aperture_mask=pld_aperture_mask,
    407     background_aperture_mask=background_aperture_mask,
    408     pld_order=pld_order,
    409     pca_components=pca_components,
    410     spline_n_knots=spline_n_knots,
    411     spline_degree=spline_degree,
    412     normalize_background_pixels=normalize_background_pixels,
    413     sparse=sparse,
    414 )
    416 clc = super().correct(
    417     dm,
    418     cadence_mask=cadence_mask,
   (...)
    421     propagate_errors=propagate_errors,
    422 )
    423 if restore_trend:

File ~/miniforge3/lib/python3.10/site-packages/lightkurve/correctors/pldcorrector.py:206, in PLDCorrector.create_design_matrix(self, pld_order, pca_components, pld_aperture_mask, background_aperture_mask, spline_n_knots, spline_degree, normalize_background_pixels, sparse)
    203 prior_sigma = np.nanstd(self.lc.flux.value) * 10
    205 # Flux normalize background components for K2 and not for TESS by default
--> 206 bkg_pixels = self.tpf.flux[:, background_aperture_mask].reshape(
    207     len(self.tpf.flux), -1
    208 )
    209 if normalize_background_pixels:
    210     bkg_flux = np.nansum(self.tpf.flux[:, background_aperture_mask], -1)

ValueError: cannot reshape array of size 0 into shape (0,newaxis)

[rebekah9969]

New tests have been added to check that

1. The code fails gracefully if all flux_err values are NaN
2. The code runs when a TPF with some NaN flux_err masked values are included.

The second test was based on the original file noted by the user which now seems to run without issue.