cmos-noise-map

This package was designed to create a read noise map for CMOS sensors, which often have pixels affected by Random Telegraph Signal.

Inputs
The expected inputs are bias images which all have the same shape.

Methods
There are a few utilities that cmos-noise-map provides, two of which are for creating a read noise map, and one of which is specifically for pixel noise modelling.

  1. std: Standard Deviation Noise Map

  2. rts: Gaussian Mixture Modeled Noise Map

  3. param: Parameter Map

Outputs If the method selected is either std or rts, the output file will be a .fits data file, in the 'READNOISE' or user specified header (by using the option --out_hdu_name)
If the method selected is param, the file will be written out as a .csv file
If the --fpacked option is used, the file will have a .fits.fz extension. We choose to compress files using fpack because it is lossless.

Standard Deviation Noise Map [Default]

The default way that the RTS map maker makes a noise map is by taking the standard deviation of the input bias images. The standard deviation is taken along the 0th axis i.e. the resultant map has the same shape of the input images, and each element is the standard deviation of a single pixel, across each of the bias images.

Input: A minimum of 10 bias frames with the same shape is required to process the images. The program will abort and exit if this is not the case.

"Three square grid blocks stacked on top of another, with a highlighted pixel in the top left corner. An arrow goes through the corner, and points to a highlighted pixel in a fourth grid block."

Above is an example of how to take the standard deviation of a pixel across images. The standard deviation of the three gray pixel values is the value recorded in the red pixel of the resultant map.

This is the simplest way to create a read noise map given a bias frame. This method is chosen as the default since it is significantly faster than modelling each pixel and calculating the read noise. The results are comparable enough that this is ok.

Running the command

Since this method is the default, we can get away with a simple command:

rts-maker <input-files-directory>

And once run, you should see a progress bar pop up, with an estimate of the time left to completion.

Note: This method does not use the upper_quantile, tolerance, or min_peak_separation options.

Whether or not your files are fpacked, the read_bias_frames function will take care of it. The files are read in with memory mapping, so theoretically we can take an arbitrarily large number of images.
Once the map is created, a fits file will be created in the path specified with the name provided. The write function does not currently have fpacking capabilities.


class cmos_noise_map.map_maker.STDMapMaker(images: list, tolerance: float = 0.05, upper_quantile: Optional[float] = None, min_peak_seperation: float = 10)
cmos_noise_map.map_maker.STDMapMaker.create_map(self)

A function to take the standard deviation of each pixel to use as a readnoise map. This is a faster method than RTSMapMaker, less rigorous statistically but achieves similar answers.

Returns:

readnoise_map – array where each element is the readnoise associated with that pixel.

Return type:

array of the same shape as the input data


The rest of the methods are NOT used in LCO production code. It is for those who are interested in the modelling methods and results.

Model Pixel Noise with GMM

The second way provided by cmos-noise-map to create a read noise map is by using a Gaussian Mixture Model to determine the read noise for pixels that are not uniformly distributed. This method is a more statistically rigorous way to determine read noise, but the algorithm is very computationally expensive. For a large CMOS sensor with a large amount of data being sampled, the process can take days. This is why this method is not the default. However, the process ideally need only be run once so users may be interested in using it.
Input: A minimum of 50 bias frames with the same shape is required to process the images. The program will abort and exit if this is not the case.

Running the command

The command is formatted as follows:

rts-maker <input-files-directory> --upper_quantile <float, optional> --tolerance <float, optional> --min_peak_separation <float, optional> <output-file-path, optional> rts

Understanding the options here is important. For more information please visit the page explaining these in detail, but in brief:

  • --upper_quantile is the noisiness cutoff for evaluating pixels. The algorithm does not evaluate every pixel, only pixels whose standard deviation is above this cutoff. By default it calculates the cutoff to be the 80th percentile of the standard deviation of all pixels. We recommend changing the parameter according to your data. Please consider using the RTS playground to determine this parameter for your data.

  • --tolerance: default=0.05 is an overfitting parameter. This is the minimum difference in the quality of fit between a model with 2 gaussians, and a model with 3. This parameter generally does not need to be changed.

  • --min_peak_separation: default=10 is the minimum separation between two modes in a multimodal pixel noise distribution, for them to be considered separate. Please consider using the RTS playground to determine this parameter for your data.

The files are read in with memory-mapping in order to not use up memory carrying around data. However, the process is multiprocessed (implemented via sklearn), so your CPU will be working hard during this process. Finally, the resultant file has the same shape as the input images, and each element is the calculated pixel read noise value. This is written out as a fits file in the path specified with the name provided.


cmos_noise_map.map_maker.MapMaker.__init__(self, images: list, tolerance: float = 0.05, upper_quantile: Optional[float] = None, min_peak_seperation: float = 10)

A class initializing the data to be passed through each map making algorithm.

Parameters:

  • images (list) – DESCRIPTION. The fits opened input images to be processed

  • tolerance (float, optional) –

    DESCRIPTION. The minimum difference between silhouette scores (likelihood of the model being correct)

    between n_components=2 and 3. If there is a plateau (i.e. not much improvement between n=2 and n=3) then the number of components chosen for the fit is 2. Not recommended to change unless you have looked at the scores yourself. The default is 0.05.

  • upper_quantile (float, optional) – DESCRIPTION. The upper standard deviation cutoff of pixels to be evaluated for telegraph noise.

  • min_peak_seperation (float, optional) – DESCRIPTION. The minimum separation of peaks for them to be considered separate peaks. If they are too close, the next lowest component is taken to be the model. The default is 10.

Return type:

None.



class cmos_noise_map.map_maker.RTSMapMaker(images: list, tolerance: float = 0.05, upper_quantile: Optional[float] = None, min_peak_seperation: float = 10)
cmos_noise_map.map_maker.RTSMapMaker.create_map(self)

A function wrapping all the methods to produce a full readnoise map

Parameters:

class (inherited from MapMaker) –

Returns:

readnoise_map – array where each element is the readnoise associated with that pixel.

Return type:

array of the same shape as the input data



cmos_noise_map.get_rts.get_rts(p, tolerance=0.05, upper_quantile=3, min_peak_separation=10)

Uses a Gaussian Mixture model, which is essentially a series of gaussian distributions of different means, variances, and amplitudes added together to model the clustering of data points. To avoid overfitting, we go through a series of logical checks to ensure that data is being fitted properly. This does NOT re-evaluate the model with every logical step, all possible fit_funs (constrained by physics) are carried through the steps.

Parameters:

  • p (TYPE: list) – DESCRIPTION: List of pixel values across a set of bias frames

  • tolerance (TYPE: float) –

    DESCRIPTION: The minimum difference between silhouette scores (likelihood of the model being correct)

    between n_components=2 and 3. If there is a plateau (i.e. not much improvement between n=2 and n=3) then the number of components chosen for the fit is 2. Not recommended to change unless you have looked at the scores yourself.

  • upper_quantile (TYPE: float) – DESCRIPTION: The upper standard deviation cutoff of pixels to be evaluated for telegraph noise.

  • min_peak_separation (TYPE: float) – DESCRIPTION: The minimum separation of peaks for them to be considered separate peaks. If they are too close, the next lowest component is taken to be the model.

Returns:

  • peak_location (TYPE: list(float), length = num_peaks) – DESCRIPTION: The means of each of the Gaussian modes calculated by GMM

  • peak_widths (TYPE: list(float), length = num_peaks) – DESCRIPTION: The covariance of each Gaussian mode calculated by GMM

  • num_peaks (TYPE: int) – DESCRIPTION: The number of Gaussians used to model the distribution of values of the pixel

  • amp (TYPE: list(float)) – DESCRIPTION: The weights of each gaussian in the mixture. All weights sum to 1.



cmos_noise_map.get_rts.readnoise(means, variances, num_peaks, amplitudes)

A function that takes model parameters for each pixel and converts it to a single readnoise value.

Parameters:

  • means (array of floats, shape (N, 1) where N is the number of model components) – The means of each gaussian used to create the model.

  • variances (array of floats, shape (N, 1) where N is the number of model components) – The covariance matrices of each gaussian used to create the model.

  • num_peaks (int) – The optimal number of peaks found by the get_rts function

  • amplitudes (list of len(N) where N is the number of model components) – The amplitude of each gaussian component modelled by get_rts.

Returns:

readnoise – The standard deviation of the pixel whose parameters are passed through, which is taken to be the read noise.

Return type:

float



cmos_noise_map.get_rts.per_pixel_readnoise(p, **kwargs)

A function wrapping get_rts and readnoise to get to the pixel readnoise

Parameters:

  • p (list) – DESCRIPTION. List of values of one pixel across images

  • **kwargs – Arguments to be passed to get_rts

Returns:

pixel_readnoise – Calculated readnoise for one pixel

Return type:

float


Getting pixel model params from GMM

This is a utility provided by the package for those who might want the parameters from our GMM modelling scheme, to do things like get more accurate estimates of photometry errors. The method is unchanged from above, we just stop short of calculating read noise, and the file written out is written as a csv file. We provide a utility function to read this csv files in, in a way that can be read by the functions in cmos-noise-map.

Running the Command

rts-maker <input-files-directory> --upper_quantile <float, optional> --tolerance <float, optional> --min_peak_separation <float, optional> <output-file-path, optional> param

All the parameters use are the same as the rts method, and parameters should be adjusted in the same way.

Below is an example of using this parameter map to get the read noise of each pixel again.

from cmos_noise_map.utils.read_write_utils import read_parameter_table
from cmos_noise_map.get_rts import readnoise

means, var, num_peaks, amps = read_parameter_table(test.csv)

readnoise_map = []
for i in range(len(means)):
    pixel_readnoise = readnoise(means[i], var[i], num_peaks[i], amps[i])
    readnoise_map.append(pixel_readnoise)

cmos_noise_map.map_maker.MapMaker.__init__(self, images: list, tolerance: float = 0.05, upper_quantile: Optional[float] = None, min_peak_seperation: float = 10)

A class initializing the data to be passed through each map making algorithm.

Parameters:

  • images (list) – DESCRIPTION. The fits opened input images to be processed

  • tolerance (float, optional) –

    DESCRIPTION. The minimum difference between silhouette scores (likelihood of the model being correct)

    between n_components=2 and 3. If there is a plateau (i.e. not much improvement between n=2 and n=3) then the number of components chosen for the fit is 2. Not recommended to change unless you have looked at the scores yourself. The default is 0.05.

  • upper_quantile (float, optional) – DESCRIPTION. The upper standard deviation cutoff of pixels to be evaluated for telegraph noise.

  • min_peak_seperation (float, optional) – DESCRIPTION. The minimum separation of peaks for them to be considered separate peaks. If they are too close, the next lowest component is taken to be the model. The default is 10.

Return type:

None.



class cmos_noise_map.map_maker.RTSParameterMapMaker(images: list, tolerance: float = 0.05, upper_quantile: Optional[float] = None, min_peak_seperation: float = 10)
cmos_noise_map.map_maker.RTSMapMaker.create_map(self)

A function wrapping all the methods to produce a full readnoise map

Parameters:

class (inherited from MapMaker) –

Returns:

readnoise_map – array where each element is the readnoise associated with that pixel.

Return type:

array of the same shape as the input data



cmos_noise_map.utils.read_write_utils.read_parameter_table(csv_file)

A utility function to read in a parameter map

Parameters:

csv_file (.csv file) – The csv file containing the parameter data output.

Returns:

  • means (TYPE: list(float), length = num_peaks) – DESCRIPTION: The means of each of the Gaussian modes calculated by GMM

  • var (TYPE: list(float), length = num_peaks) – DESCRIPTION: The covariance of each Gaussian mode calculated by GMM

  • num_peaks (TYPE: int) – DESCRIPTION: The number of Gaussians used to model the distribution of values of the pixel

  • amps (TYPE: list(float)) – DESCRIPTION: The weights of each gaussian in the mixture. All weights sum to 1.


Main wrapper function


cmos_noise_map.main.cli(*args: Any, **kwargs: Any) → Any

This script builds a noise map with the chosen method.

path: Path to input bias files

filepath:OPTIONAL Path to write file. Default writes it in the current directory.

method: Default method is std. Available methods are std, rts, and param. See docs for more information about each method.

Base functions to calculate the read noise from standard deviation


cmos_noise_map.get_rts.get_rts(p, tolerance=0.05, upper_quantile=3, min_peak_separation=10)

Uses a Gaussian Mixture model, which is essentially a series of gaussian distributions of different means, variances, and amplitudes added together to model the clustering of data points. To avoid overfitting, we go through a series of logical checks to ensure that data is being fitted properly. This does NOT re-evaluate the model with every logical step, all possible fit_funs (constrained by physics) are carried through the steps.

Parameters:

  • p (TYPE: list) – DESCRIPTION: List of pixel values across a set of bias frames

  • tolerance (TYPE: float) –

    DESCRIPTION: The minimum difference between silhouette scores (likelihood of the model being correct)

    between n_components=2 and 3. If there is a plateau (i.e. not much improvement between n=2 and n=3) then the number of components chosen for the fit is 2. Not recommended to change unless you have looked at the scores yourself.

  • upper_quantile (TYPE: float) – DESCRIPTION: The upper standard deviation cutoff of pixels to be evaluated for telegraph noise.

  • min_peak_separation (TYPE: float) – DESCRIPTION: The minimum separation of peaks for them to be considered separate peaks. If they are too close, the next lowest component is taken to be the model.

Returns:

  • peak_location (TYPE: list(float), length = num_peaks) – DESCRIPTION: The means of each of the Gaussian modes calculated by GMM

  • peak_widths (TYPE: list(float), length = num_peaks) – DESCRIPTION: The covariance of each Gaussian mode calculated by GMM

  • num_peaks (TYPE: int) – DESCRIPTION: The number of Gaussians used to model the distribution of values of the pixel

  • amp (TYPE: list(float)) – DESCRIPTION: The weights of each gaussian in the mixture. All weights sum to 1.



cmos_noise_map.get_rts.readnoise(means, variances, num_peaks, amplitudes)

A function that takes model parameters for each pixel and converts it to a single readnoise value.

Parameters:

  • means (array of floats, shape (N, 1) where N is the number of model components) – The means of each gaussian used to create the model.

  • variances (array of floats, shape (N, 1) where N is the number of model components) – The covariance matrices of each gaussian used to create the model.

  • num_peaks (int) – The optimal number of peaks found by the get_rts function

  • amplitudes (list of len(N) where N is the number of model components) – The amplitude of each gaussian component modelled by get_rts.

Returns:

readnoise – The standard deviation of the pixel whose parameters are passed through, which is taken to be the read noise.

Return type:

float



cmos_noise_map.get_rts.per_pixel_readnoise(p, **kwargs)

A function wrapping get_rts and readnoise to get to the pixel readnoise

Parameters:

  • p (list) – DESCRIPTION. List of values of one pixel across images

  • **kwargs – Arguments to be passed to get_rts

Returns:

pixel_readnoise – Calculated readnoise for one pixel

Return type:

float