Image classification of night time images taken from the International Space Station

Alejandro Sánchez de Miguel, et al. Universidad Complutense de Madrid.

Image classification:

Right now there are around 1.800.000 images at the Johnson Space Center database (The Gateway of the Astronauts). Around 1.200.000 images were taken aboard the ISS (date 20/02/2014). However the number of the classified images is much smaller and there is no archive of georeferenced images. There is a project to classify the day time images (Image detective). But, the techniques that are used in this project are not useful for the classification of night time images. The reason is that the patterns on Earth are not the same during the day and night. This is why another technique is needed to classify these night time, images.

Our main objective it's to study the light pollution that came from the cities. We want to stop the waste of energy and the destruction of the mighty ecosystem.

Moreover, on the research of night time images they also interesting those containing star fields (Zamorano et al. 2011).

Automatic classification techniques of images:

Classification based on the ISS nadir position:

The vertical of the ISS it is a point called Nadir. This position is easily calculated approximately using the orbit of the ISS and the time when the image was taken. Also, the JSC gives this information sometimes in their database.

Although, these positions are only a first approximation and the place that have been captured can be beyond 300 km (187 miles). This effect can produce false positives or negatives depending on the tolerance to the moment selected for the instant of the sunset or the sunrise.

This position can be used to classify the images taken from the ISS. Based on the fact that if the Sun is over the horizon it is not a night time image. Also it can be interesting to know if the Moon is also over the horizon and its phase. In order to make this calculus the python package pyephem can be useful. We must take into account that the time on the EXIF information of the images is on Universal Time (UT).

The first attempt to compute the image position is to know if ISS nadir is on the night side of the earth. This is computed by iss.py and posicionISS.py script. They use the TLE orbital elements and PyEphem package. TLE files are provided since 2009 to 2014.

Classification based on the statistics of the image (histogram):

This technique was explored by J. Gómez Castaño and A. Sánchez de Miguel, cited by M. López Cayuela 2012 Degree Thesis (2012) – in Spain. This technique is based on the great number of dark pixels and histogram distribution on the night time images. Although, this technique produce a lot of false positives when the astronauts take pictures near the horizon during the day time or pictures inside the ISS or non-terrestrial images. Also, they can produce false negatives ones when high resolution images are taken. The high resolution images of big cities don’t present places without population, so there are no dark places, but the histogram distribution provide positive when near 0 values are found.

Histogram distribution script is shown in histograma.py file. This is provided to developer to allow them include it into their final software

After the automated classification it is need it a manual classification.

Figure 1 Histogram of a night time image. The mode it’s a bit low, a round 11.

Figure 2 Histogram of a day time image. It's clear that the mode it is clearly higher that on the night image. A round 111 counts.

We need to distinguish between these categories.

Images of Earth with cities.

Figure 3 Example of night time images of cities.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS038/ISS038-E-41063.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS038/ISS038-E-44501.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS038/ISS038-E-10481.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS038/ISS038-E-44520.jpg

Night time images of earth without cities.

Figure 4 Example of night time images without cities.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS034/ISS034-E-45243.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-137690.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-130127.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-158151.jpg

Images of stars:

Figure 5 Example of images with stars.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-149149.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-6460.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-5619.jpg

Images of auroras:

Figure 6 Example of an image of an aurora.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-116196.jpg

Day time images of earth.

Figure 7 Examples of day time images.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS038/ISS038-E-512.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS038/ISS038-E-20373.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS038/ISS038-E-28104.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-5411.jpg

Infrared images.

Figure 8 Example of infrared images. They are clearly more red than normal images.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-41672.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-41549.jpg

Moon and other celestial bodies.

Figure 9 Example of images of the moon from the ISS.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-170022.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-275771.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-147343.jpg

Sunsets and sunrises:

Figure 10 Images of the sunset or sun rise from the ISS.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-31485.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-37642.jpg

Inside the ISS:

Figure 11 Examples of images taken inside the ISS.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-7551.jpg

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-32252.jpg

Other (combination previous categories).

Figure 12 Example of one image with stars, infrared and storms.

http://earth.jsc.nasa.gov/sseop/images/ESC/small/ISS030/ISS030-E-21177.jpg

Stars, infrared, cities, and storms.

Georeferenciation:

First step.

As we have explained on the introduction, based on the time that the picture was taken and the coordinates given from the JSC we can know approximately where the images were taken. Although, the astronauts don’t shoot only to their nadir. So, sometimes the center of the images can be more than 300 km (186 miles) from their nadir. That’s why we have to inspect around to find the center of the image. Because the maps and the day time images are very different of to the night time, we will use the georeference night images of the satellite SUOMI-NPP and its instrument VIIRS/DNB.

Figure 13 Example of the precatalog coordinates from the JSC.

Figure 14 Example of the image of the SUOMI-NPP VIIRS/DNB on the Google Earth Galery. Center on the saame coordinates of the figure 13.

http://eol.jsc.nasa.gov/scripts/sseop/photo.pl?mission=ISS030&roll=E&frame=51845

https://mapsengine.google.com/10446176163891957399-13737975182519107424-4/mapview/?version=2&hl=es

Full georeferenciation.

We need to make a high quality geolocation, so we need no change the shape of the image. So we will capture the geographic coordinates that correspond to some coordinates X and Y of the image. On the zoom out or low resolution images we will use the night layer, but on the high resolution ones we will use the day time images or the street maps.

Final Software requirements: To allow a full geolocation for a image, we need to get at least 4 points that can be fit to a map. The requirements are as follows:

The user interface have to allow the image can be rotated to adjust to the correct orientation.

The user interface have to include zoom in, zoom out and pan options.

A base map, preferable a ortophotographic one, have to be place as a layer below the image. So the user can rotate it and translate it to adjust it to the real position and zoom

Once the image is on the correct location, a control has to allow get 4 points over it. The system will get these points and its fit coordinates. The X, Y, longitude, latitude values, have to be stored into a database table, including the image reference.

Figure 15 Example of control points using the SUOMI Image

Figure 16 Example of control points using the google maps street map.

Other interesting aspects of the classification.

Also there are some other aspects that can be interesting to take into account. As can be the presence of clouds or the sharpness of the images.

Figure 17 Blurred image of Madrid but clear.

Figure 18 Clouds on the 30% of the image buts it is a very sharp image.

Fields that should be on the database:

ID of the classification: Number of classification. Example: 10000.

ID of the image on the ISSNight database. Example: ISS030-E-51845

ID of the IMAGE: ID of the image on the JSC database. Example: ISS030-E-51845

Nadir JSC: The nadiris the direction pointing directly below a particular location. Based on the JSC information. Example: Nadir Latitude: 42.8 Nadir Longitude: -86.5

Nadir TIME: The nadiris the direction pointing directly below a particular location. Based on the time of the camera and the orbit of the ISS.

Country: Country we is the centre of the center of the image.

Place: The name of the nearest recognizable place on the image.

Orientation: Where is the north on the image.

Comments: Particular things characteristics of the image. Presence of reflections or other things.

Link to the image: Link to the original JPG on the JSC web.

Author of the classification: User ID.

Quality: sharpness of the image.

Lens: AF Nikkor 50mm f/1.4D

Focal length: 50.0 mm

Camera model. NIKON D3S S/N: 2008336

Shutter.Example:1/40

Aperture. Example:1.4

ISO. Example:10000

Already classified by JSC: Boolean.

Other references: Twitter, new or papers. Example:

https://twitter.com/PC0101/status/435784019024703488/photo/1

Tilt: Example: 50º

Tags: Tags corresponding to the normalized classification. Example: Cities at Night.

Control points Image (pixels): Coordinates on X and Y. Example: 200.39, 300.89

Control points on earth (decimal degrees): Geographical coordinates of the image Control points. Example: 40.365, -3.765

Sun Elevation Angle: -60 (Angle in degrees between the horizon and the sun, measured at the nadir point)

GMT Date: 20120129 (YYYYMMDD) GMT Time: 071621 (HHMMSS)

First attempt application

A first attempt application is provided to participants. This is a set of python scripts which can connect to JSC ftp site, download images, analyze them and create a SQL script to inject the image information into a MySQL database.

The following SQL insert sentence is provided as an example

INSERT INTO astro.imagenesISS (id, fecha, imagen, focal, camara, exposicion, apertura, iso, pais, lugar, tilt, orientacion, latitudcentro, longitudcentro, H) VALUES (NULL, '2011-10-13 12:07:14', 'ISS029-E-25846.JPG','17.0 mm','NIKON D3S S/N: 2007934','1','3.2','12800','','','','','','','5.0');

Resources:

Website of JSC:

http://eol.jsc.nasa.gov/

URL large images: http://eol.jsc.nasa.gov/sseop/images/ESC/large/ISS026/ISS026-E-35953.JPG

URL small images: http://eol.jsc.nasa.gov/sseop/images/ESC/small/ISS023/ISS023-E-29061.JPG

URL of the thumbs: http://eol.jsc.nasa.gov/sseop/images/thumb/ISS030/ISS030-E-9660.jpg

URL of the record card of the image:

http://eol.jsc.nasa.gov/scripts/sseop/photo.pl?mission=ISS030&roll=E&frame=51845

URL exif data: http://eol.jsc.nasa.gov/sseop/camera/ISS030/ISS030-E-51845.txt

Useful libraries:

Pyephem: http://rhodesmill.org/pyephem/

References:

Zamorano, J., de Miguel, A. S., Pascual, S., Castaño, J. G., Ramírez, P., & Challupner, P. (2011). ISS nocturnal images as a scientific tool against Light Pollution. LICA report, April.

Zamorano, J., de Miguel, A. S., Ocaña, F., & Castaño, J. G. (2013). El uso de imágenes de satélite para combatir la contaminación lumínica. Astronomía, (167), 80-87.

Sánchez de Miguel, A., Zamorano, J. M., Gómez Castaño, J., Ocaña, F., Pascual Ramírez, S., López Cayuela, M. A., ... & Challupner, P. (2013, May). Contaminación lumínica en España 2012: Light pollution in Spain 2012. In Highlights of Spanish Astrophysics VII, Proceedings of the X Scientific Meeting of the Spanish Astronomical Society (SEA), held in Valencia, July 9-13, 2012, Eds.: JC Guirado, LM Lara, V. Quilis, and J. Gorgas., pp. 956-956 (Vol. 1, pp. 956-956).

Sánchez de Miguel, A., Zamorano, J., Pascual, S., López Cayuela, M., Ocaña, F., Challupner, P., ... & de Miguel, E. (2013, May). ISS nocturnal images as a scienti c tool against Light Pollution: Flux calibration and colors. In Highlights of Spanish Astrophysics VII, Proceedings of the X Scientific Meeting of the Spanish Astronomical Society (SEA), held in Valencia, July 9-13, 2012, Eds.: JC Guirado, LM Lara, V. Quilis, and J. Gorgas., pp. 916-919 (Vol. 1, pp. 916-919).

Zamorano Calvo, J., Sánchez de Miguel, A., Gómez Castaño, J., Ocaña González, F., Gallego Maestro, J., Pila Díez, B., ... & Pascual Ramírez, S. (2013). Night Sky Brightness and Light Pollution in Comunidad de Madrid.

Sánchez de Miguel, A.(2013), Variation of the Sky brightness with the phase and height of the moon.