In a recent post titled Unweaving the rainbow, Matt Hall described our joint attempt (partly successful) to create a Python tool to enable recovery of digital data from any pseudo-colour scientific image (and a seismic section in particular, like the one in Figure 1), without any prior knowledge of the colormap.
Figure 1. Test image: a photo of a distorted seismic section on my wall.
Please check our GitHub repositoryfor the code and slides andwatch Matt’s talk (very insightful and very entertaining) from the 2017 Calgary Geoconvention below:
In the next two post, coming up shortly, I will describe in greater detail my contribution to the project, which focused on developing a computer vision pipeline to automatically detect where the seismic section is located in the image, rectify any distortions that might be present, and remove all sorts of annotations and trivia around and inside the section. The full workflow is included below (with sections I-VI developed to date):
I – Image preparation, enhancement:
Convert to gray scale
Optional: smooth or blur to remove high frequency noise
Enhance contrast
II – Find seismic section:
Convert to binary with adaptive or other threshold method
Find and retain only largest object in binary image
Fill its holes
Apply opening and dilation to remove minutiae (tick marks and labels)
III – Define rectification transformation
Detect contour of largest object find in (2). This should be the seismic section.
Approximate contour with polygon with enough tolerance to ensure it has 4 sides only
Sort polygon corners using angle from centroid
Define new rectangular image using length of largest long and largest short sides of initial contour
Estimate and output transformation to warp polygon to rectangle
IV – Warp using transformation
V – Blanking annotations inside seismic section (if rectangular):
Start with output of (4)
Pre-process and apply canny filter
Find contours in the canny filter smaller than input size
Sort contours (by shape and angular relationships or diagonal lengths)
Loop over contours:
Approximate contour
If approximation has 4 points AND the 4 semi-diagonals are of same length: fill contour and add to mask
VI – Use mask to remove text inside rectangle in the input and blank (NaN) the whole rectangle.
VII – Optional: tools to remove arrows and circles/ellipses:
For arrows – contours from (4) find ones with 7 sizes and low convexity (concave) or alternatively Harris corner and count 7 corners, or template matching
For ellipses – template matching or regionprops
VIII – Optional FFT filters to remove timing lines and vertical lines
You can download from GitHub all the tools for the automated workflow (parts I-VI) in the module mycarta.py, as well as an example Jupyter Notebook showing how to run it.
The first post focuses on the image pre-processing and enhancement, and the detection of the seismic line (sections I and II, in green); the second one deals with the rectification of the seismic (sections IV to V, in blue). They are not meant as full tutorials, rather as a pictorial road map to (partial) success, but key Python code snippets will be included and discussed.
This guest post (first published here) is by Elwyn Galloway, author of Scibbatical on WordPress. It is the forth in our series of collaborative articles about sketch2model, a project from the 2015 Calgary Geoscience Hackathonorganized by Agile Geoscience. Happy reading.
As Matteo demonstrated by example, sketch2model’s ability to segment a sketch properly depends on the fidelity of a sketch.
An image of a whiteboard sketch (left) divides an area into three sections. Without morphological filtering, sketch2model segments the original image into two sections (identified as orange, purple) (centre). The algorithm correctly segments the area into three sections (orange, purple, green) when morphological filtering is applied (right).
To compensate for sketch imperfections, Matteo suggested morphological filtering on binarized images. Morphological filtering is a set of image modification tools which modify the shape of elements in an image. He suggested using the closing tool for our purposes. Have a look at Matteo’s Post for insight into this and other morphological filters.
One of the best aspects of this approach is that it is simple to apply. There is essentially one parameter to define: a structuring element. Since you’ve already read Matteo’s post, you recall his onion analogy explaining the morphological filtering processes of erosion and dilation – erosion is akin to removing an onion layer, dilation is adding a layer on. You’ll also recall that the size of the structuring element is the thickness of the layer added to, or removed from, the onion. Essentially, the parameterization of this process comes down to choosing the thickness of the onion layers.
Sketch2model uses dilation followed by erosion to fill gaps left between sketch lines (morphological dilation followed by erosion is closing). Matteo created this really great widget to illustrate closing using an interactive animation.
Matteo’s animation was created using this interactive Jupyter notebook. Closing connects the lines of the sketch.
Some is Good, More is Better?
Matteo showed that closing fails if the structural element used is too small. So just make it really big, right? Well, there can be too much of a good thing. Compare what happens when you use an appropriately sized structuring element (mild) to the results from an excessively large structuring element (wild).
Comparing the results of mild and wild structuring elements: if the structuring element is too large, the filter compromises the quality of the reproduction.
Using a morphological filter with a structural element that is too small doesn’t fix the sketches, but using a structural element that is too large compromises the sketch too. We’re left to find an element that’s just right. Since one of the priorities for sketch2model was to robustly handle a variety of sketches with as little user input as possible — marker on whiteboard, pencil on paper, ink on napkin — we were motivated to find a way to do this without requiring the user to select the size of the structuring element.
Is there a universal solution? Consider this: a sketch captured in two images, each with their own resolution. In one image, the lines of the sketch appear to be approximately 16 pixels wide. The same lines appear to be 32 pixels wide in the other image. Since the size of the structuring element is defined in terms of pixels, it becomes apparent the ideal structuring element cannot be “one size fits all”.
High-resolution (left) versus low-resolution (right) image of the same portion of a sketch. Closing the gap between the lines would require a different size structuring element for each image: about 5 pixels for high-resolution or 1 pixel for low-resolution.
Thinking Like a Human
Still motivated to avoid user parameterization for the structuring element, we explored ways to make the algorithm intelligent enough to select an appropriate structuring element on its own. Ultimately, we had to realize a few things before we came up with something that would work:
When capturing an image of a sketch, users compose very similar images (compose in the photographic sense of the word): sketch is centered and nearly fills the captured image.
The image of a sketch is not the same as a user’s perception of a sketch: a camera may record imperfections (gaps) in a sketch that a user does not perceive.
The insignificance of camera resolution: a sketched feature in captured at two different resolutions would have two different lengths (in pixels), but identical lengths when defined as a percentage of image size.
With these insights, we deduced that the gaps we were trying to fill with morphological filtering would be those that escaped the notice of the sketch artist.
Recognizing the importance of accurate sketch reproduction, our solution applies the smallest structuring element possible that will still fill any unintentional gaps in a sketch. It does so in a way that is adaptable.
A discussion about the definition of “unintentional gap” allowed us to create a mandate for the closing portion of our algorithm. Sketch2model should fill gaps the user doesn’t notice. The detail below the limit of the user’s perception should not affect the output model. A quick “literature” (i.e. Google) search revealed that a person’s visual perception is affected by many factors beyond the eye’s optic limits. Without a simple formula to define a limit, we did what any hacker would do… define it empirically. Use a bunch of test images to tweak the structuring element of the closing filter to leave the perceptible gaps and fill in the imperceptible ones. In the sketch2model algorithm, the size of structuring element is defined as a fraction of the image size, so it was the fraction that we tuned empirically.
Producing Usable Results
Implicit in the implementation is sketch2model’s expectation that the user’s sketch, and their image of the sketch are crafted with some care. The expectations are reasonable: connect lines you’d like connected; get a clear image of your sketch. Like so much else in life, better input gives better results.
Input (left) and result (right) of sketch2model.
To produce an adaptable algorithm requiring as little user input as possible, the sketch2model team had to mix a little image processing wizardry with some non-technical insight.
In the first post of this series I showed how to use Pandas, Seaborn, and Matplotlib to:
load a dataset
test, clean up, and summarize the data
start looking for relationships between variables using scatterplots and correlation coefficients
In this second post, I will expand on the latter point by introducing some tests and visualizations that will help highlight the possible criteria for choosing some variables, and dropping others. All in Python.
The target to be predicted is oil production from a marine barrier sand. We have measured production (in tens of barrels per day) and 7 unknown (initially) predictors, at 21 wells.
Hang on tight, and read along, because it will be a wild ride!
I will show how to:
1) automatically flag linearly correlated predictors, so we can decide which might be dropped. In the example below (a matrix of pair-wise correlation coefficients between variables) we see that X2, and X7, the second and third best individual predictors of production (shown in the bottom row) are also highly correlated to X1, the best overall predictor.
2) automatically flag predictors that fail a critical r test
3) create a table to assess the probability that a certain correlation is spurious, in other words the probability of getting at least the correlation coefficient we got with our the sample, or even higher, purely by chance.
I will not recommend to run these tests and apply the criteria blindly. Rather, I will suggest how to use them to learn more about the data, and in conjunction with domain knowledge about the problem at hand (in this case oil production), make more informed choices about which variables should, and which should not be used.
And, of course, I will show how to make the prediction.
In a previous post I showed some of the beautiful new images of Pluto from New Horizon’s mission, coloured using the new Matplotlib perceptual colormaps:
More recently I was experimenting with Principal Component Analysis in scikit-learn, and one of the things I used it for was compression of some of these Pluto images. Below is an example of the first two components from the False Color Pluto image:
The idea behind this series of articles is to show how to predict P-wave velocity, as measured by a geophysical well log (the sonic), from a suite of other logs: density, gamma ray, and neutron, and also depth, using Machine Learning.
I will explore different Machine Learning methods from the scikit-learn Python library and compare their performances.
To wet your appetites, here’s an example of P-wave velocity, Vp, predicted using a cross-validated linear model, which will be the benchmark for the performance of other models, such as SVM and Random Forest:
In the first notebook, which is already available on GitHub here, I show how to use the Pandas and Seaborn Python libraries to import the data, check it, clean it up, and visualize to explore relationships between the variables. For example, shown below is a heatmap with the pairwise Spearman correlation coefficient between the variables (logs):
Stay tuned for the next post / notebook!
PS: I am very excited by the kick-off of the Geophysical Tutorial (The Leading Edge) Machine Learning Contest 2016. Check it out here!
At 11:53 am of September 14 2015, Marco Drago, an Italian postdoctoral scientist at the Max Planck Institute for Gravitational Physics (AKA Albert Einstein Institute) in Hannover, Germany, was the first person to see it: the sophisticated instruments at the Laser Interferometer Gravitational-Wave Observatory (LIGO) had detected a very likely candidate signal from gravitational waves. The signal, caused by the collision and merger of two massive black holes, proved to be real; the discovery, announced to the public a few months later, demonstrated one of Albert Einstein’s predictions exactly 100 years after he formulated his general theory of relativity. How exciting!
Most of us have studied that gravity produces a curvature in spacetime. A less generally known, but major prediction of general relativity is that a perturbation of spacetime produces gravitational waves, which are detected because they stretch and squeeze space producing a measurable strain; this however requires a violent cosmological phenomenon involving large masses, relativistic speeds (approaching the speed of light), and asymmetrical acceleration (an abrupt change in those speeds, such as in a collision).
One such event is that which generated the signal detected in September of last year. It occurred about 1300 million years ago when two large black holes spiralled towards one another and then merged into a single, stationary black hole, losing energy by way of radiating gravitational waves (ripples in space-time) exactly as predicted by Einstein; the whole process took only a few tenths of a second, but involved a total of 65 solar masses (36 + 29), 3 of which were converted, mostly in the final instant of the merger, to gravitational wave energy according to the famous relationship E = mc2. This was so much energy that, had it been visible (electromagnetic), it would’ve been 50 times brighter than the entire universe. Yet, even a huge event like that created very small waves (gravity is the weakest of the four fundamental interactions of nature), which merely displaced a pair of free-falling masses placed 3 km apart by a length 10,000 smaller than the diameter of a proton. Measuring this displacement requires very complex arrays of laser beams, mirrors, and detectors measuring interference (interferometers), such as the 2 LIGO in the United States, and VIRGO in Italy.
This video illustrates in detail how VIRGO (and LIGO) is able to detects the very weak waves; it was produced by Marco Kraan of the National Institute for Subatomic Physics in Amsterdam, and shown during Carbognani’s talk:
The figure below shows an illustration of the event’s 3 main phases and the corresponding, matched signals from the 2 LIGO interferometers.
LIGO detects gravitational waves from merging black holes. Illustration credit: LIGO, NSF, Aurore Simonnet (Sonoma State University).
Python’s many contributions
Franco Carbognani is VIRGO integration manager with the European Gravitational Observatory in Pisa. The portion of his talk focusing on Python and VIRGO’s control systems starts here. He told the audience that where Python played a major role was in the building of a complete automation layer on top of real-time interferometer control, with analysis of data online and warning to the operator in case of anomalies, and also in GUI development and unification. Python was the obvious choice for these tasks because it is a compact, clear language, easy for beginners to learn, and yet allowing very complex programming (functional, object oriented, imperative, exceptions, etc.); its Numpy and Scipy libraries allow handling of the complex math required, without sacrificing speed (thanks to the optimized Fortran and C under the hood); a large collection of other libraries allows for almost any task to be carried out, including (as mentioned above) Matplotlib for publication quality graphs; finally, it is open-source, and many in the community already used it (commissioning, computing, and data analytics groups).
Tito dal Canton is a postdoctoral scientist at the Max Planck Institute in Hannover. The data analysis part of his talk starts here. The workflow he outlined, involving the use of several separate pipelines, consists of retrieving the data, deciding which data can be analyzed, decide if it contains a signal, estimate its statistical significance, and estimating the signal parameters (e.g. masses of stars, spin velocity, and distance) by comparison with a model. A lot of this work is run entirely in Python using either the GWpy and PyCBC packages. For the keener reader, one of the Jupyter Notebooks on the LIGO Python tutorials page, replicates some of the signal processing and data analysis shown during his talk.
As written by Elwyn in the first post of this series, sketch2model was conceived at the 2015 Calgary Geoscience Hackathon as a web and mobile app that would turn an image of geological sketch into a geological model, and then use Agile Geoscience’s modelr.io to create a synthetic seismic model.
The skech2model concept: modelling at the speed of imagination. Take a sketch (a), turn it into an earth model (b), create a forward seismic model (c). Our hack takes you from a to b.
One of the main tasks in sketch2model is to identify each and every geological body in a sketch as a closed polygon. As Elwyn wrote, “if the sketch were reproduced exactly as imagined, a segmentation function would do a good job. The trouble is that the sketch captured is rarely the same as the one intended – an artist may accidentally leave small gaps between sketch lines, or the sketch medium can cause unintentional effects (for example, whiteboard markers can erase a little when sketch lines cross, see example below). We applied some morphological filtering to compensate for the sketch imperfections.
Morphological filtering can compensate for imperfections in a sketch, as demonstrated in this example. The original sketch (left) was done with a marker on white board. Notice how the vertical stroke erased a small part of the horizontal one. The binarized version of the sketch (middle) shows an unintentional gap between the strokes, but morphological filtering successfully closes the small gap (right).
The cartoon below shows what would be the final output of sketch2model in the two cases in the example above (non closed and closed gap).
My objective with this post is to explain visually how we correct for some of these imperfections within sketch2model. I will focus on the use of morphological closing, which consist in applying in sequence a dilation and an erosion, the two fundamental morphological operations.
Quick mathematical morphology review
All morphological operations result from the interaction of an image with a structuring element (a kernel) smaller than the image and typically in the shape of a square, disk, or diamond. In most cases the image is binary, that is pixels take either value of 1, for the foreground objects, or 0 for the background. The structuring element operates on the foreground objects.
Morphological erosion is used to remove pixels on the foreground objects’ boundaries. How ‘deeply’ the boundaries are eroded depends on the size of the structuring element (and shape, but in this discussion I will ignore the effect of changing the shape). This operation is in my mind analogous to peeling off a layer from an onion; the thickness of the layer is related to the structuring element size.
Twan Maintz in his book Digital and medical image processingdescribes the interaction of image and structuring element during erosion this way: place the structuring element anywhere in the image: if it is fully contained in the foreground object (or in one of the objects) then the origin (central) pixel of the structuring element (and only that one) is part of the eroded output. The book has a great example on page 129.
Dilation does the opposite of erosion: it expands the object boundaries (adding pixels) by an amount that is again related to the size of the structuring element. This is analogous to me to adding back a layer to the onion.
Again, thanks to Maintz the interaction of image and structuring element in dilation can be intuitively described: place the structuring element anywhere in the image: does it touch any of the foreground objects? If yes then the origin of the structuring element is part of the dilated result. Great example on pages 127-128.
Closing is then for me akin to adding a layer to an onion (dilation) and then peeling it back off (erosion) but with the major caveat that some of the changes produced by the dilation are irreversible: background holes smaller than the structuring element that are filled by the dilation are not restored by the erosion. Similarly, lines in the input image separated by an amount of pixels smaller than the size of the structuring element are linked by the dilation and not disconnected by the erosion, which is exactly what we wanted for sketch2model.
Closing demo
If you still need further explanation on these morphological operations, I’d recommend reading further on the ImageMagik user guide the sections on erosion, dilation, and closing, and the examples on the Scikit-image website.
As discussed in the previous section, when applying closing to a binary image, the external points in any object in the input image will be left unchanged in the output, but holes will be filled, partially or completely, and disconnected objects like edges (or lines in sketches) can become connected.
I will use this model binary image containing two 1-pixel wide lines. Think of them as lines in a sketch that should have been connected, but are not.
We will attempt to connect these lines using morphological closing with a disk-shaped structuring element of size 2. The result is plotted in the binary image below, showing that closing was successful.
But what would have happened with a smaller structuring element, or with a larger one? In the case of a disk of size 1, the closing magic did not happen:
Observing this result, one would increase the size of the structuring element. However, as Elwyn will show in the next post, also too big a structuring element would have detrimental effects, causing subsequent operations to introduce significant artifacts in the final results. This has broader implications for our sketch2model app: how do we select automatically (i.e. without hard coding it into the program) the appropriate structuring element size? Again, Elwyn will answer that question; in the last section I want to concentrate on explaining how the closing machinery works in this case.
In the next figure I have broken down the closing operation into its component dilation and erosion, and plotted them step by step to show what happens:
So we see that the edges do get linked by the dilation, but by only one pixel, which the following erosion then removes.
And now let’s break down the closing with disk of size two into its component. This is equivalent to applying two consecutive passes of dilation with disk of size 1, and then two consecutive passes of erosion with disk of size 1, as in the demonstration in the next figure below (by the way, if we observed carefully the second panel above we could predict that the dilation with a disk of size two would result in a link 3-pixel wide instead of 1-pixel wide, which the subsequent erosion will not disconnect).
Below is a GIF animated version of this demo, cycling to the above steps; you can also run it yourself by downloading and running the Jupyter notebook on GitHub.
