Data Acquisition

Timestamped Video Source

Module for video source acquisition. Classes capture data from a video source into a numpy array.

class sksurgeryimage.acquire.video_source.TimestampedVideoSource(source_num_or_file, dims=None)[source]

Bases: object

Capture and store data from camera/file source. Augments the cv2.VideoCapture() to provide passing of camera dimensions in constructor, and storage of frame data.

grab()[source]

Call the cv2.VideoCapture grab function and get a timestamp.

isOpened()[source]

Call the cv2.VideoCapture isOpened function.

read()[source]

Do a grab(), then retrieve() operation.

release()[source]

Release the cv2.VideoCapture source.

retrieve()[source]

Call the cv2.VideoCapture retrieve function and store the returned frame.

set_resolution(width: int, height: int)[source]

Set the resolution of the input source.

Parameters:
  • width (int) – Width
  • height (int) – Height
Raises:

ValueError – If resolution is not supported.

class sksurgeryimage.acquire.video_source.VideoSourceWrapper[source]

Bases: object

Wrapper for multiple TimestampedVideoSource objects.

add_camera(camera_number, dims=None)[source]

Create VideoCapture object from camera and add it to the list of sources.

Parameters:
  • camera_number – integer camera number
  • dims – (width, height) as integer numbers of pixels
add_file(filename, dims=None)[source]

Create videoCapture object from file and add it to the list of sources.

Parameters:
  • filename – a string containing a valid file path
  • dims – (width, height) as integer numbers of pixels
add_source(camera_num_or_file, dims=None)[source]
Add a video source (camera or file) to the list of sources.
Parameters:
  • camera_num_or_file – either an integer camera number or filename
  • dims – (width, height) as integer numbers of pixels
are_all_sources_open()[source]

Check all input sources are active/open.

get_next_frames()[source]

Do a grab() operation for each source, followed by a retrieve().

grab()[source]

Perform a grab() operation for each source

release_all_sources()[source]

Close all camera/file sources.

retrieve()[source]

Perform a retrieve operation for each source. Should only be run after a grab() operation.

:returns list of views on frames

Stereo Video Source

Module for stereo video source acquisition.

class sksurgeryimage.acquire.stereo_video.StereoVideo(layout, channels, dims=None)[source]

Bases: object

Provides a convenient object to manage various stereo input styles. Developed firstly for laparoscopic surgery, but broadly applicable to any stereo setup using our TimestampedVideoSource and VideoSourceWrapper.

Design Principles:

  1. Fail early, throwing exceptions for all errors.
  2. Works with or without camera parameters.
  3. If no camera parameters, calling get_undistorted() or get_rectified() is an Error.
get_images()[source]

Returns the 2 channels, unscaled, as a list of images.

Returns:list of images
get_rectified()[source]

Returns the 2 channels, rectified, as a list of images.

Returns:list of images
Raises:ValueError, TypeError - if camera parameters are not set.
get_scaled()[source]

Returns the 2 channels, scaled, as a list of images.

Returns:list of images
get_undistorted()[source]

Returns the 2 channels, undistorted, as a list of images.

Returns:list of images
Raises:ValueError - if you haven’t already provided camera parameters
grab()[source]

Asks internal VideoSourceWrapper to grab images.

release()[source]

Asks internal VideoSourceWrapper to release all sources.

retrieve()[source]

Asks internal VideoSourceWrapper to retrieve images.

set_extrinsic_parameters(rotation, translation, dims)[source]

Sets the stereo extrinsic parameters.

Parameters:
  • rotation – 3x3 numpy array representing rotation matrix.
  • translation – 3x1 numpy array representing translation vector.
  • dims – new image size for rectification
Raises:

ValueError, TypeError

set_intrinsic_parameters(camera_matrices, distortion_coefficients)[source]

Sets both sets of intrinsic parameters.

Parameters:
  • camera_matrices – list of 2, 3x3 numpy arrays.
  • distortion_coefficients – list of 2, 1xN numpy arrays.
Raises:

ValueError, TypeError

class sksurgeryimage.acquire.stereo_video.StereoVideoLayouts[source]

Bases: object

Class to hold some constants, like an enum.

DUAL = 0
INTERLACED = 1
VERTICAL = 2

Video Writing

Write stream of frames to file using OpenCV

class sksurgeryimage.acquire.video_writer.ThreadedTimestampedVideoWriter(filename, fps=25, width=640, height=480, codec='MJPG')[source]

Bases: sksurgeryimage.acquire.video_writer.TimestampedVideoWriter

TimestampedVideoWriter that can be run in a thread. Uses Queue.Queue() to store data, which is thread safe.

Frames will be processed as they are added to the queue:

threaded_vw = ThreadedTimestampedVideoWriter(file, fps, w, h) threaded_vw.start()

threaded_vw.add_to_queue(frame, timestamp) threaded_vw.add_to_queue(frame, timestamp) threaded_vw.add_to_queue(frame, timestamp)

threaded_vw.stop()

run()[source]

Write data from the queue to the output file(s).

start()[source]

Start the thread running.

stop()[source]

Stop thread running.

write_frame(frame, timestamp=None)[source]

Add a frame and a timestamp to the queue for writing. Named for consistency with the non-threaded version. Actual writing to disk is done by write_frame_to_disk() :param frame: Image frame :type frame: numpy array :param timestamp: Frame timestamp :type timestamp: datetime.datetime object

write_frame_to_disk()[source]

Get frame and timestamp from queue, then write to output.

class sksurgeryimage.acquire.video_writer.TimestampedVideoWriter(filename, fps=25, width=640, height=480, codec='MJPG')[source]

Bases: sksurgeryimage.acquire.video_writer.VideoWriter

Class to write images and timestamps to disk, inherits from VideoWriter.

Parameters:
  • fps – Frames per second to save to disk.
  • filename – Filename to save output video to. Timestamp file is “filename + ‘timestamps’”
close()[source]

Close/release the output files for video and timestamps.

write_frame(frame, timestamp=None)[source]

Write a frame and timestamp to the output files. If no timestamp provided, write a defualt value. :param frame: Image data :type frame: numpy array :param timestamp: Timestamp data :type timestamp: datetime.datetime object

class sksurgeryimage.acquire.video_writer.VideoWriter(filename, fps=25, width=640, height=480, codec='MJPG')[source]

Bases: object

Class to write images to disk using cv2.VideoWriter.

Parameters:
  • fps – Frames per second to save to disk.
  • filename – Filename to save output video to.
  • width – width of input frame
  • height – height of input frame
check_valid_filename(filename)[source]

Return true if filename is a string.

close()[source]

Close/release the output file for video.

create_output_dir_if_needed()[source]

Check if the directory specified in file path exists and create if not.

set_filename(filename)[source]

Set the filename to write to.

write_frame(frame)[source]

Write a frame to the output file.

Calibration Tools

Point Detector

Base class for a PointDetector.

e.g. Chessboard corners, SIFT points, Charuco points etc.

class sksurgeryimage.calibration.point_detector.PointDetector(scale=(1, 1), camera_intrinsics=None, distortion_coefficients=None)[source]

Bases: object

Class to detect points in a 2D video image.

These point detectors are often used to detect points for camera calibration. However, it would also be possible for some subclasses to utilise camera intrinsics and distortion coefficients in order to improve the point detection process itself. It would be up to the derived class to decide how to use them, if at all.

Parameters:
  • scale – tuple (x scale, y scale) to scale up/down the image
  • camera_intrinsics – [3x3] camera matrix
  • distortion_coefficients – [1xn] distortion coefficients
get_camera_parameters()[source]

Returns a copy of the camera matrix, and distortion coefficients. Throws RuntimeError if either are None.

Returns:[3x3], [1xn] matrices
get_model_points()[source]

Derived classes should override this, to detector returns the complete model of 3D points. e.g. for a chessboard this would be all the corners in chessboard coordinates (e.g. z=0).

By design, this can return an ndarray with zero rows, if the detector does not support 3D coordinates.

Returns:[Nx3] numpy ndarray representing model points.
get_points(image, is_distorted=True)[source]

Client’s call this method to extract points from an image.

Parameters:
  • image – numpy 2D RGB/grayscale image.
  • is_distorted – False if the input image has already been undistorted.
Returns:

ids, object_points, image_points as Nx[1,3,2] ndarrays

set_camera_parameters(camera_intrinsics, distortion_coefficients)[source]

Enables camera parameters to be set dynamically at run-time. Calls _validate_camera_parameters().

Parameters:
  • camera_intrinsics – [3x3] camera matrix
  • distortion_coefficients – [1xn] distortion coefficients

Chessboard Detector

Chessboard implementation of PointDetector.

class sksurgeryimage.calibration.chessboard_point_detector.ChessboardPointDetector(number_of_corners, square_size_in_mm, scale=(1, 1))[source]

Bases: sksurgeryimage.calibration.point_detector.PointDetector

Class to detect chessboard points in a 2D grey scale video image.

get_model_points()[source]

Returns a [Nx3] numpy ndarray representing the model points in 3D.

ArUco Point Detector

ArUco implementation of PointDetector.

class sksurgeryimage.calibration.aruco_point_detector.ArucoPointDetector(dictionary, parameters, model, scale=(1, 1))[source]

Bases: sksurgeryimage.calibration.point_detector.PointDetector

Class to detect ArUco points in a 2D grey scale video image.

Note: For ArUco points, these don’t have to be on a regular grid. If you provide a ‘model’ which is a map of id : 3D point, the function _internal_get_points will provide the corresponding 3D points of those points that were detected.

get_model_points()[source]

Returns a [Nx3] numpy ndarray representing the model points in 3D.

sksurgeryimage.calibration.aruco_point_detector.get_intersect(a_1, a_2, b_1, b_2)[source]

Returns the point of intersection of the lines passing through a2,a1 and b2,b1.

See https://stackoverflow.com/questions/3252194/numpy-and-line-intersections

Parameters:
  • a_1 – [x, y] a point on the first line
  • a_2 – [x, y] another point on the first line
  • b_1 – [x, y] a point on the second line
  • b_2 – [x, y] another point on the second line

ChArUco Point Detector

ChArUco implementation of PointDetector.

class sksurgeryimage.calibration.charuco_point_detector.CharucoPointDetector(dictionary, number_of_squares, size, scale=(1, 1), camera_matrix=None, distortion_coefficients=None, filtering=False)[source]

Bases: sksurgeryimage.calibration.point_detector.PointDetector

Class to detect ChArUco points in a 2D video image.

get_model_points()[source]

Returns a [Nx3] numpy ndarray representing the model points in 3D.

ChArUco & Chesboard Detector

ChArUco + Chessboard implementation of PointDetector.

class sksurgeryimage.calibration.charuco_plus_chessboard_point_detector.CharucoPlusChessboardPointDetector(reference_image, minimum_number_of_points=50, scale=(1, 1), number_of_charuco_squares=(19, 26), size_of_charuco_squares=(5, 4), dictionary=<aruco_Dictionary 0x7fb1e4743670>, camera_matrix=None, distortion_coeff=None, charuco_filtering=False, use_chessboard_inset=True, number_of_chessboard_squares=(9, 14), chessboard_square_size=3, chessboard_id_offset=500, error_if_no_chessboard=True, error_if_no_charuco=False)[source]

Bases: sksurgeryimage.calibration.point_detector.PointDetector

Class to detect ChArUco points and Chessboard points in a 2D grey scale video image.

get_model_points()[source]

Returns a [Nx3] numpy ndarray representing the model points in 3D.

Dotty Grid Point Detector

Dotty Grid implementation of PointDetector.

class sksurgeryimage.calibration.dotty_grid_point_detector.DottyGridPointDetector(model_points, list_of_indexes, camera_intrinsics, distortion_coefficients, scale=(1, 1), reference_image_size=None, rms=30, gaussian_sigma=5, threshold_window_size=151, threshold_offset=20, min_area=50, max_area=50000, dot_detector_params=None)[source]

Bases: sksurgeryimage.calibration.point_detector.PointDetector

Class to detect a grid of dots in a 2D grey scale video image.

More specifically, a grid of dots with 4 larger dots at known locations.

get_model_points()[source]

Returns a [Nx3] numpy ndarray representing the model points in 3D.

sksurgeryimage.calibration.dotty_grid_point_detector.get_model_points(dots_rows_columns: (<class 'int'>, <class 'int'>), pixels_per_mm: int, dot_separation: float) → numpy.ndarray[source]

Generate the expected locations of dots in the pattern, in pixel space.

Parameters:
  • dots_rows_columns ([int, int]) – Number of rows, number of columns
  • pixels_per_mm (int) – Pixels per mm
  • dot_separation (float) – Distance between dots in mm
Returns:

array pf point info - [id, x_pix, y_pix, x_mm, y_mm, z_mm]

Return type:

np.ndarray

ChArUco Helper Functions

Functions to support camera calibration using ChArUco chessboard markers.

sksurgeryimage.calibration.charuco.detect_charuco_points(dictionary, board, image, camera_matrix=None, distortion_coefficients=None, filtering=False)[source]

Extracts ChArUco points. If you can provide camera matrices, it may be more accurate.

Parameters:
  • dictionary – aruco dictionary definition
  • board – aruco board definition
  • image – grey scale image in which to search
  • camera_matrix – if specified, the 3x3 camera intrinsic matrix
  • distortion_coefficients – if specified, the distortion coefficients
  • filtering – if True, filter out wrongly detected markers
Returns:

marker_corners, marker_ids, chessboard_corners, chessboard_ids

sksurgeryimage.calibration.charuco.draw_charuco_corners(image, chessboard_corners, chessboard_ids)[source]

Function to draw chessboard corners on an image.

Parameters:
  • image – input image
  • chessboard_corners – from detect_charuco_points
  • chessboard_ids – from detect_charuco_points
Returns:

new image with corners marked

sksurgeryimage.calibration.charuco.erase_charuco_markers(image, marker_corners)[source]

Method to automatically blank out ChArUco markers, leaving an image that looks like it contains just a chessboard, rather than ChArUco board. It does this by drawing a plain white polygon, with vertices defined by the tag detection process. So, on a synthetic image, this works perfectly. On a real image, due to blurring or other artefacts such as combing, there may be some residual.

Parameters:
  • image – image containing a view of a ChArUco board.
  • marker_corners – detected corners
Returns:

edited image

sksurgeryimage.calibration.charuco.filter_out_wrong_markers(marker_corners, marker_ids, board)[source]

Filters out markers that were mis-labelled. For each inner corner on the ChArUco board, if both neighbouring markers are detected, look at the projected positions of this corner using the perspective transformations obtained form the two markers. If the two positions are not close (further than 20 pixels away), then at least one of the markers is mis-labelled but we won’t know which one. Remove both markers.

Parameters:
  • marker_corners – marker corners detected by OpenCV
  • marker_ids – ids of markers detected
  • board – charuco board definition
Returns:

marker_corners, marker_ids

sksurgeryimage.calibration.charuco.make_charuco_board(dictionary, number_of_squares, size, image_size)[source]

Generates a ChArUco pattern.

Don’t forget to select an image size that is a nice multiple of the square size in millimetres, to avoid any interpolation artefacts. You should check the resultant image has only 2 values, [0|255], and nothing interpolated between these two numbers.

Parameters:
  • dictionary – aruco dictionary definition
  • number_of_squares – tuple of (number in x, number in y)
  • size – tuple of (size of chessboard square, size of internal tag), mm.
  • image_size – tuple of (image width, image height), pixels.
Returns:

image, board

sksurgeryimage.calibration.charuco.make_charuco_with_chessboard(dictionary=<aruco_Dictionary 0x7fb1e48cbe30>, charuco_squares=(19, 26), charuco_size=(5, 4), pixels_per_millimetre=10, chessboard_squares=(9, 14), chessboard_size=3, chessboard_border=0.7)[source]

Helper function to make an image of a calibration target combining ChArUco markers and a chessboard. It’s up to the caller to work out a nice number of pixels per millimetre, so that the resultant image is correctly scaled.

Defaults are as used in SmartLiver project. Not also, that we compute the image and coordinates in portrait, but it’s used in landscape.

Parameters:
  • dictionary – ChArUco dictionary
  • charuco_squares – tuple of (squares in x, squares in y)
  • charuco_size – tuple of (external size, internal tag size) in mm
  • pixels_per_millimetre – which determines size of eventual image.
  • chessboard_squares – tuple of (squares in x, squares in y)
  • chessboard_size – size of chessboard squares in mm
  • chessboard_border – border round chessboard, as fraction of square
Returns:

calibration image

Point Detector Utils

Utilities, specific to the PointDetector stuff.

sksurgeryimage.calibration.point_detector_utils.write_annotated_image(input_image, ids, image_points, image_file_name)[source]

Takes an input image, copies it, annotates point IDs and writes to the testing output folder.

Utilities

Camera Utilities

Functions to check cameras.

sksurgeryimage.utilities.camera_utilities.count_cameras()[source]

Count how many camera sources are available. This is done by trying to instantiate cameras 0..9, and presumes they are in order, sequential, starting from zero.

Returns:int, number of cameras
sksurgeryimage.utilities.camera_utilities.validate_camera_input(camera_input)[source]

Checks that camera_input is an integer, and it is a valid camera.

Param:camera_input, integer of camera

Video Interlacing Functions

Functions to support deinterlacing, reinterlacing and vertical destacking of 2D video frames.

sksurgeryimage.processing.interlace.deinterlace_to_new(interlaced)[source]

Takes the interlaced image, and splits into two new images of even_rows and odd_rows.

Returns:even_rows, odd_rows images
sksurgeryimage.processing.interlace.deinterlace_to_preallocated(interlaced, even_rows, odd_rows)[source]

Deinterlaces the interlaced image into even_rows and odd_rows images, which must be pre-allocated, and the correct size.

sksurgeryimage.processing.interlace.deinterlace_to_view(interlaced)[source]

Takes the interlaced image, and returns two new views of even_rows and odd_rows.

Returns:even_rows, odd_rows images
sksurgeryimage.processing.interlace.interlace_to_new(even_rows, odd_rows)[source]

Interlaces even_rows and odd_rows images into a new output image.

sksurgeryimage.processing.interlace.interlace_to_preallocated(even_rows, odd_rows, interlaced)[source]

Interlaces even_rows and odd_rows images into the interlaced image, where all inputs must be pre-allocated to the correct size.

sksurgeryimage.processing.interlace.split_stacked_to_new(stacked)[source]

Takes the input stacked image, and extracts the top and bottom half.

Useful if you have stereo 1080x1920 inputs, into an AJA Hi5-3D which stacks them vertically into 2 frames of 540x1920 in the same image.

Returns:top_half and bottom_half images
sksurgeryimage.processing.interlace.split_stacked_to_preallocated(stacked, top, bottom)[source]

Splits a vertically stacked image, extracting the top and bottom halves, assuming images are the right size and pre-allocated.

Useful if you have stereo 1080x1920 inputs, into an AJA Hi5-3D which stacks them vertically into 2 frames of 540x1920 in the same image.

sksurgeryimage.processing.interlace.split_stacked_to_view(stacked)[source]

Takes the input stacked image, and returns views that refer to the top and bottom half.

Returns:top_half, bottom_half images
sksurgeryimage.processing.interlace.stack_to_new(left, right)[source]

Vertically stack left and right array into single output array. Left and right images should have the same dimensions.

Parameters:
  • left (numpy array) – left image
  • right (numpy array.) – right image
sksurgeryimage.processing.interlace.validate_interlaced_image_sizes(even_rows, odd_rows, interlaced)[source]

Validates the sizes of the even_rows, odd_rows and interlaced images.

  1. Inputs must all be numpy images.
  2. Inputs must all have the same number of columns.
  3. Inputs must all have an even number of rows.
  4. even_rows and odd_rows must have the same number of rows.
  5. even_rows and odd_rows must have half the number of rows as interlaced.

Morphological Operators

Functions to support morphological operators.

In many cases, these will just be convenience wrappers around OpenCV functions.

sksurgeryimage.processing.morphological_operators.dilate_with_cross(src, dst=None, size=3, iterations=1)[source]

Dilates an image with a cross element. OpenCV supports both grey scale and RGB erosion.

Parameters:
  • src – source image
  • dst – if provided, an image of the same size as src
  • size – size of structuring element
  • iterations – number of iterations
Returns:

the eroded image

sksurgeryimage.processing.morphological_operators.erode_with_cross(src, dst=None, size=3, iterations=1)[source]

Erodes an image with a cross element. OpenCV supports both grey scale and RGB erosion.

Parameters:
  • src – source image
  • dst – if provided, an image of the same size as src
  • size – size of structuring element
  • iterations – number of iterations
Returns:

the eroded image

Image Cropper

Misc

Various utilities, like preparing overlay text.

sksurgeryimage.utilities.utilities.are_similar(image0, image1, threshold=0.995, metric=5, mean_threshold=0.005)[source]

Compares two images to see if they are similar.

Parameters:
  • image0 (image0,) – The images
  • threshold – The numerical threshold to use, default 0.995
  • method – The comparison metric, default normalised cross correlation, cv2.TM_CCOEFF_NORMED
  • mean_threshold – Also compare the mean values of each array, return false if absolute difference of image means divided by the average of both images is greater than the mean_threshold, if less than zero this test will be skipped
Returns:

True if the metric is greater than the threshols, false otherwise or if the images are not the same dimensions or type

sksurgeryimage.utilities.utilities.image_means_are_similar(image0, image1, threshold=0.005)[source]

Compares two images to see if they have similar mean pixel values

Parameters:
  • image0 (image0,) – The images
  • threshold – The mean value threshold to use. return false if absolute difference of image means divided by the average of both images is greater than the mean_threshold.
Returns:

false if absolute difference of image means divided by the average of both images is greater than the mean_threshold, true otherwise or if threshold is less than zero.

sksurgeryimage.utilities.utilities.noisy_image(image, mean=0, stddev=(50, 5, 5))[source]

Creates a noise image, based on the dimensions of the passed image. param: the image to define size and channels of output returns: a noisy image

sksurgeryimage.utilities.utilities.prepare_cv2_text_overlay(overlay_text, frame, text_scale=1)[source]

Return settings for text overlay on a cv2 frame.

A class for making a natty WEISS logo

Bases: object

Creates a WEISS logo and passes a copy on request

Returns the WEISS Logo

Returns:The WEISS Logo as a Numpy array

Returns the WEISS Logo with some noise added

Returns:A noisy WEISS Logo as Numpy Array
sksurgeryimage.utilities.weisslogo.circle(img, center, radius, color[, thickness[, lineType[, shift]]]) → img

. @brief Draws a circle. . . The function cv::circle draws a simple or filled circle with a given center and radius. . @param img Image where the circle is drawn. . @param center Center of the circle. . @param radius Radius of the circle. . @param color Circle color. . @param thickness Thickness of the circle outline, if positive. Negative values, like #FILLED, . mean that a filled circle is to be drawn. . @param lineType Type of the circle boundary. See #LineTypes . @param shift Number of fractional bits in the coordinates of the center and in the radius value.

sksurgeryimage.utilities.weisslogo.fillConvexPoly(img, points, color[, lineType[, shift]]) → img

. @brief Fills a convex polygon. . . The function cv::fillConvexPoly draws a filled convex polygon. This function is much faster than the . function #fillPoly . It can fill not only convex polygons but any monotonic polygon without . self-intersections, that is, a polygon whose contour intersects every horizontal line (scan line) . twice at the most (though, its top-most and/or the bottom edge could be horizontal). . . @param img Image. . @param points Polygon vertices. . @param color Polygon color. . @param lineType Type of the polygon boundaries. See #LineTypes . @param shift Number of fractional bits in the vertex coordinates.