Interactive Deep Learning-Based Retinal OCT Layer Segmentation Refinement by Regressing Translation Maps

Obtaining a precise and personalized layer boundary still requires physicians to actively correct different regions on multiple layers.

“Because of this, obtaining a precise and personalized layer boundary still requires physicians to actively correct different regions on multiple layers [16].”

• cites [16] sourcePartly supported

  Section 4.3.2 (p. 12) says expert annotators can correct model-generated masks interactively and that volumetric images involve delineation across multiple slices. It does not establish that precise, personalized layer boundaries necessarily require physicians to correct different regions on multiple layers. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Retinal layer segmentation has an associated subjectivity, with specialists often preferring different but valid solutions in severe pathological regions.

“Also, retinal layer segmentation, similarly to other medical segmentation tasks, has an associated subjectivity, with specialists often preferring different but valid solutions in severe pathological regions.”

• Also stated in: § Abstract · plain

  “nor for the specialists' subjectivity.”

• cites [16] sourcePartly supported

  Section 5.3 (p. 27) says medical-image segmentation labels can differ between readers because of human subjectivity about ambiguous lesions. It does not establish that specialists often prefer different but valid retinal-layer segmentations specifically, or that this occurs in severe pathological regions. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Anatomically coherent layer ordering is guaranteed by the method as in [15].

“Anatomically coherent layer ordering is guaranteed as in [15]:”

• cites [15] sourceSupported

Pixel-wise segmentation approaches tend to underperform on retinal boundary segmentation in OCT images because they do not properly encode anatomical priors such as layer ordering, allowing segmented layers to occur at unreasonable locations.

“Indeed, pixel-wise segmentation approaches tend to underperform on retinal boundary segmentation in OCT images as they do not properly encode anatomical priors such as layer ordering, allowing segmented layer to occur at unreasonable locations [15].”

• cites [15] sourceSupported

On the DME dataset, the method from He et al. [15] achieved an average absolute distance of 6.70 μm.

“Average | 12.01 6.29 | 7.37 0.93 | 7.34 1.08 | 6.70 | 6.74 4.10”

• cites [15] sourcePartly supported

  The Experiments section describes the DME dataset, and “Surface position prediction accuracy” says the method’s boundary mean absolute distance (MAD) is reported in Table 2. The passages available from our copy do not give the Table 2 value, so they do not verify the claimed average of 6.70 μm. (LLM, conf=0.77)

  Experimental — accuracy not yet measured

Interacting with a single boundary affects neighboring segmentations.

“As shown, interacting with a single boundary affects neighboring segmentations.”

The error reduction with both simulated and real interactions is very similar.

“In particular, we show that the error reduction with both simulated and real interactions is very similar.”

Preliminary experiments showed no meaningful gain on using mappings other than the binary matrix for encoding user interactions.

“Although a common approach is to process these inputs with 2D Gaussian kernels of different spreads [27], preliminary experiments showed no meaningful gain on using mappings other than the binary matrix.”

• cites [27] sourceNot checked — no full text of the source

The behavior of the network is similar across datasets with different types of retinal diseases.

“In addition, the behavior of the network is similar across datasets with different types of retinal diseases.”

A single click per layer improves the segmentation in a pathological region.

“This is also quantitatively shown in Figs.6b and 6c, where a single click per layer improves the segmentation in a pathological region.”

The approach achieves state-of-the-art performance on multiple private and public datasets with different pathomorphological complexities.

“We extensively validate our approach on multiple private and public datasets with different pathomorphological complexities, achieving state-of-theart performance, while allowing for a simple and efficient user interaction.”

• Also stated in: § B. CONTRIBUTIONS · plain

  “a novel deep learning approach for retinal boundary segmentation, that not only achieves a performance similar or better than other state-of-the-art approaches, but importantly, also allows for interactive correction.”

• Also stated in: § B. CONTRIBUTIONS · plain

  “an extensive quantitative and qualitative validation of the approach on private and public datasets spanning different retinal diseases, and of different pathomorphological complexities.”

The proposed method achieved a performance similar or better than other state-of-the-art approaches on both the HC/MS and DME datasets.

“From the tables, we can observe that our method achieved a performance similar or better than other state-of-the-art approaches on both datasets.”

The average error of the baseline is still much larger than that of the proposed solution.

“However, the average error is still much larger than that of the proposed solution.”

Initial experiments showed that very high values of w_l near the correction regions lead the network to overly focus on those locations, degrading the segmentation performance elsewhere.

“Initial experiments also showed that very high values of w l near the correction regions lead the network to overly focus on those locations, degrading the segmentation performance elsewhere.”

The error reduction with interactions tends to be less stable.

“Furthermore, as shown in Figs. 5 a, b and c, the error reduction with interactions tends to be less stable.”

For the HC/MS dataset, the method had the best score on 4 out of the 9 boundaries (RMSE metric).

“For the HC/MS dataset, specifically, we had the best score on 4 out of the 9 boundaries (RMSE metric).”

Both real users were capable of achieving an overall better performance than the simulation.

“Indeed, both real users were capable of achieving an overall better performance than the simulation.”

Using our approach allows a better mitigation of existing segmentation errors in comparison to manual corrections.

“Using our approach allows a better mitigation of existing segmentation errors in comparison to manual corrections.”

For all datasets, the upper left region of the plot has high error, corresponding to segmentation regions with low confidence and low number of interactions.

“Indeed, for all datasets the upper left region of the plot has high error, which is to be expected as it corresponds to segmentation regions with low confidence and low number of interactions.”

The assisted interaction allows to obtain smooth transitions between the corrected and neighbour regions of the boundary, resulting in a more natural segmentation than the manual correction.

“Also, the assisted interaction allows to obtain smooth transitions between the corrected and neighbour regions of the boundary, resulting in a more natural segmentation than the manual correction, as exemplified in Fig. 7e, where the length of the stroke is smaller than the corrected depressions of the OS-RPE layer.”

The approach is effective, as demonstrated by extensive quantitative and qualitative validations in both private and public datasets.

“We demonstrate the effectiveness of the approach by conducting extensive quantitative and qualitative validations in both private and public datasets.”

• AssumptionThe authors state this as an assumption: User interactions are always meaningful, i.e. would always lead to a better segmentation.
• AssumptionRests on an assumption: Fully manual correction to measure retinal layer thickness is not feasible due to severely limited time availability of retinal specialists and high amount of work.

The system is adjusting to the user's opinion, moving the boundary towards the requested locations.

“Instead, the system is adjusting to the user's opinion, moving the boundary towards the requested locations.”

The approach allows correcting the initial segmentations.

“and, more importantly, to correct those segmentations.”

For extreme outlier clicks, the segmentation behaviour is suboptimal.

“For extreme outlier clicks (see Fig. 7f), the segmentation behaviour is suboptimal.”

Larger segmentation errors, in particular for the IPL-INL, INL-OPL and OPL-ONL, are corrected with a few clicks.

“Note how larger segmentation errors, in particular for the IPL-INL, INL-OPL and OPL-ONL (see Fig 6b, left half), are corrected with a few clicks (Fig. 6f).”

These clicks lead to updates mainly on the same neighborhood region of the translation map, allowing to maintain the quality of the segmentation on the remaining portions of the image.

“Internally, these clicks lead to updates mainly on the same neighborhood region of the translation map, allowing to maintain the quality of the segmentation on the remaining portions of the image.”

The approach, based on predicting translation maps, achieves an initial segmentation performance in line with other state-of-the-art methods.

“Our approach, based on predicting translation maps, allows obtaining an initial segmentation performance in line with other state-of-the-art methods (Tables 1 and 2)”

If all layers are to be corrected, the assisted correction is particularly efficient in comparison to manual annotations when very small (single clicks and small strokes) are provided.

“If all layers are to be corrected, the assisted correction is particularly efficient in comparison to the manual annotations when very small (single clicks and small strokes) are provided.”

A few user-provided clicks in regions with large error substantially improve the initial segmentation.

“We showed that a few user provided clicks in the regions with large error allow to substantially improve the initial segmentation (Fig. 6).”

• AssumptionThe authors state this as an assumption: User interactions are always meaningful, i.e. would always lead to a better segmentation.

Correcting one of the boundaries locally improved the quality of the neighboring segmentations.

“Likewise, Fig. 7d and 7e show scenarios where correcting one of the boundaries locally improved the quality of the neighboring segmentations.”

The model stops actively contributing to a better correction when the input strokes are larger than approximately 2% of the images' width (approx. 10 to 20 pixels, depending on the dataset).

“Specifically, the model stops actively contributing to a better correction when the input strokes are larger than approximately 2% of the images' width (approx.10 to 20 pixels, depending on the dataset).”

Training to account for an expected layer ordering allows correcting multiple layers while interacting with just one.

“In addition, training to account for an expected layer ordering allows correcting multiple layers while interacting with just one, as exemplified for example in Fig. 7e.”

• AssumptionThe authors state this as an assumption: User interactions are always meaningful, i.e. would always lead to a better segmentation.

In the DME dataset, there is a set of cases for which increasing the number of clicks degrades the quality of the segmentation (first column of plot (a), corresponding to 0-0.25 confidence).

“Note, however, that in the DME dataset (Fig. 10(a)), there is a set of cases for which increasing the number of clicks degrades the quality of the segmentation (first column of plot (a), corresponding to 0-0.25 confidence).”

The system has an associated uncertainty estimation.

“Furthermore, our system has an associated uncertainty estimation (Fig. 10)”

The system learned to rely on the user input to produce different outputs according to the opinion of the annotator, thus accounting for the subjectivity inherent to the segmentation process.

“The system also learned to rely on the user input to produce different outputs according to the opinion of the annotator, thus accounting for the subjectivity inherent to the segmentation process (Fig. 8).”

• AssumptionThe authors state this as an assumption: User interactions are always meaningful, i.e. would always lead to a better segmentation.

On the DME dataset, the proposed method achieved an average absolute distance of 6.74 μm (standard deviation 4.10 μm).

“Average | 12.01 6.29 | 7.37 0.93 | 7.34 1.08 | 6.70 | 6.74 4.10”

None of the interactions matches the reference segmentation.

“Notice that none of the interactions matches the reference segmentation.”

User interaction allows the method to achieve segmentation performance beyond the current state-of-the-art.

“In particular, user interaction allows the method to achieve the segmentation performance beyond the current state-of-the-art.”

• AssumptionThe authors state this as an assumption: User interactions are always meaningful, i.e. would always lead to a better segmentation.
• AssumptionRests on an assumption: Fully manual correction to measure retinal layer thickness is not feasible due to severely limited time availability of retinal specialists and high amount of work.

The system is capable of correcting layer boundary segmentations in scans covering three different retinal diseases with minimal annotation effort.

“Indeed, the system was shown capable of correcting layer boundary segmentations in scans covering three different retinal diseases with minimal annotation effort (Fig. 5).”

• AssumptionThe authors state this as an assumption: User interactions are always meaningful, i.e. would always lead to a better segmentation.

After a single poorly located click, the click only affects its respective column.

“Unlike the cases shown in Fig. 6, where clicks made on plausible locations affect the correction of the adjacent portions of the boundaries, in this case the click only affects its respective column.”

The results with real interactions validate the findings using simulated interactions.

“These results validate our findings using the simulated interactions.”

The model outputs a pixel-wise confidence map of how likely it is that a specific boundary occurs on that location, easing the identification of regions where the segmentation is likely to need correction.

“In addition, the model outputs a pixelwise confidence map (Fig. 2, Confidence map) of how likely it is that a specific boundary occurs on that location, easing the identification of regions where the segmentation is likely to need correction.”

The proposed approach combines the strengths of pixel- and contour-focused approaches by treating the layer boundary segmentation task as a pixel-wise regression problem where the network predicts the amount of movement required to properly readjust the coordinates of the current segmentation.

“The approach combines the strengths of pixel-and contour-focused approaches by treating the layer boundary segmentation task as a pixel-wise regression problem where the network has to predict the amount of movement required to properly readjust the coordinates of the current segmentation.”

The proposed method does not directly output a boundary, but instead translates it to new positions.

“Our method doesn't directly output a boundary, but instead translates it to new positions.”

The low inference time facilitates the translation of the system to the clinic.

“The low inference time per prediction iteration of approx.0.25s/2.5s(GPU/CPU, independent of the number of interactions performed), further facilitates the translation of the system to the clinic.”

The contour-focused approach allows operating on the final predictions' coordinates in an end-to-end fashion, easing the addition of anatomical constraints that improve the local segmentation behavior.

“Instead, our contourfocused approach allows to operate on the final predictions' coordinates in an end-to-end fashion, easing the addition of anatomical constraints that improve the local segmentation behavior.”

The system outputs a coordinate-wise confidence, allowing to automatically identify regions of possible segmentation failure that may require user attention.

“Additionally, the system outputs a coordinate-wise confidence, allowing to automatically identify regions of possible segmentation failure that may require user attention.”

• Also stated in: § B. CONTRIBUTIONS · plain

  “an interactive system that outputs a coordinate-wise confidence, allowing to automatically identify regions of possible segmentation failure that may require user attention”

At each prediction iteration, the model accounts for user feedback to predict how much a point anywhere on the image would have to move to be in the correct location.

“Ultimately, at each prediction iteration, the model accounts for user feedback to predict how much a point anywhere on the image would have to move to be in the correct location.”

The method is designed to allow for segmentation correction by interactions with point-clicks or line-scribbles.

“The method is designed to allow for segmentation correction by interactions with point-clicks or line-scribbles.”

• Also stated in: § B. CONTRIBUTIONS · plain

  “The approach is specifically designed to allow for segmentation correction by simple user interactions with point clicks or line scribbles”

The model infers for each layer boundary a pixel-wise translation map with the estimation of the signed vertical distance to the corresponding boundary coordinate.

“The model infers for each layer boundary a pixel-wise translation map (Fig. 2, Translation map) with the estimation of the signed vertical distance to the corresponding boundary coordinate.”

This leads to a coherent progression of the segmentation without requiring costly and slow parameter tuning via retraining.

“This leads to a coherent progression of the segmentation without requiring costly and slow parameter tuning via, for instance, retraining.”

The simulated interaction, which follows the training scheme by always selecting the highest error coordinate, may not be the best interaction strategy.

“In fact, it appears that the simulated interaction, which follows the training scheme by always selecting the highest error coordinate, may not even be the best interaction strategy.”

The model interprets users' clicks and scribbles to adapt the translation maps T so that the boundaries move to the desired new location.

“The model interprets users' clicks and scribbles to adapt the translation maps T so that the boundaries move to the desired new location.”

The system is robust to different types/styles of interactions.

“Also, the similar performance of both manual annotations suggests that the system is robust to different types/styles of interactions.”

The approach was extensively validated quantitatively and qualitatively on private and public datasets spanning different retinal diseases and pathomorphological complexities.

“an extensive quantitative and qualitative validation of the approach on private and public datasets spanning different retinal diseases, and of different pathomorphological complexities.”

• Also stated in: § Abstract · boosted

  “We extensively validate our approach on multiple private and public datasets with different pathomorphological complexities, achieving state-of-theart performance, while allowing for a simple and efficient user interaction.”

The baseline model has learned to account for the spatial location of the user clicks to improve the segmentation.

“The behavior of the baseline indicates that the model has learned to account for the spatial location of the user clicks to improve the segmentation.”

The deep learning-based approach predicts the translation required to correct layer boundary segmentations by regressing pixel-wise translation maps that account for user input.

“Our deep learning-based approach predicts the translation required to correct layer boundary segmentations by regressing pixel-wise translation maps that account for the user input.”

• Also stated in: § B. CONTRIBUTIONS · plain

  “we propose a new segmentation and interaction scheme that predicts the translation required to correct segmentations by regressing pixel-wise translation maps that account for the user input.”

• Also stated in: § II. METHODS · plain

  “The model infers for each layer boundary a pixel-wise translation map (Fig. 2, Translation map) with the estimation of the signed vertical distance to the corresponding boundary coordinate.”

As long as access to the initially predicted coordinates is possible, the approach allows training an interactive correction method that can be used by clinicians to refine segmentations according to their needs.

“As long as access to the initially predicted coordinates is possible, our approach allows training an interactive correction method that can be used by clinicians to refine segmentations according to their needs.”

• AssumptionRests on an assumption: Fully manual correction to measure retinal layer thickness is not feasible due to severely limited time availability of retinal specialists and high amount of work.

The system can account for user subjectivity.

“Fig. 8 illustrates how the system can account for user subjectivity.”

The model has learned to leverage the confidence as a weighting factor on the loss function to reduce the influence of high error locations during training.

“Results indicate that the model has learned to leverage the confidence as a weighting factor on the loss function to reduce the influence of high error locations during training.”

This behavior is expected because the model was specifically trained to perform corrections based on clicks and not large strokes.

“This behaviour is to be expected, as the model was specifically trained to perform corrections based on clicks and not large strokes, with the latter being between the simulated manual user interaction and the deep learning assisted interaction as function of the number of interactions and length of the correction stroke (percentage of image width).”

A segmentation proposed by a different system can be used as the starting point for interaction, because the correction method is based on the translation of existing boundaries instead of their direct prediction.

“Of note, a segmentation proposed by a different system can be used as the starting point for interaction, because the correction method is based on the translation of existing boundaries instead of their direct prediction.”

The sparse interaction encoding and the extra loss weight near the interaction regions promote a local segmentation correction without degrading the performance elsewhere.

“Particularly, the sparse interaction encoding and the extra loss weight near the interaction regions promote a local segmentation correction without degrading the performance elsewhere.”

• AssumptionThe authors state this as an assumption: User interactions are always meaningful, i.e. would always lead to a better segmentation.

Click-based interactions are of particular interest for regions where multiple boundaries have an initial poor estimation, as they significantly reduce the amount of manual effort required for correction.

“Click-based interactions are of particular interest for regions where multiple boundaries have an initial poor estimation, as they significantly reduce the amount of manual effort required for correction (Fig. 6).”

The paper presents a novel interactive segmentation method for retinal boundary segmentation in OCT images.

“This paper presents a novel interactive segmentation method for retinal boundary segmentation in OCT images.”

• Also stated in: § V. DISCUSSION AND CONCLUSION · plain

  “In conclusion, a novel interactive segmentation approach for retinal boundary segmentation in OCT images is presented.”

• AssumptionRests on an assumption: Fully manual correction to measure retinal layer thickness is not feasible due to severely limited time availability of retinal specialists and high amount of work.