KML4Earth


Reference

© Copyright 2011 jasonm1

logoKML Best Practice  

The increase in applications consuming and producing KML (which powers applications like Google Earth™) requires that best practices be established to promote interoperability. This document describes best practices for validating KML and for using metadata and attribution. These best practices supplement the OGC KML 2.2 & 2.3 Specifications and Google KML Reference.

Validating KML

The importance of validating KML has been demonstrated by the rapid growth of applications other than Google Earth™ that consume and produce KML. Google has implemented extreme flexibility in its tools relative to KML in order to enable easy and widespread use of Google's Mapping products such as Google Earth. However, from a validation perspective, this flexibility equates to lax validation of KML so it can accept KML loosely, adhering to the OGC KML Specification and corresponding KML XML Schema. This is not a defect in Google Earth, but it should illustrate that Google Earth is not intended to be a KML validation tool and other tools are needed to perform validation if you want the KML to view the same in whatever tool reads it.

Basically this means that having data appear in Google Earth may give a false sense that the same data will appear like that elsewhere so additional tools are needed for validation.

As more and more applications in addition to the Google Earth™ client consume KML it is even more important to validate the KML produced. The lax parsing of loosely-defined KML may give KML producers the impression that their KML is valid when it is not. In fact, even if you turn on error checking in Google Earth you may only be prompted if an unknown tag/namespace is encountered or a tag is in the wrong context or the KML file is an invalid XML file (i.e., not well-formed).

Since then Google has taken a stricter stance on conforming to the KML specification in Google Earth. This decision deliberately broke backwards-compatibility for some badly formatted KML. This is a step forward, but the client doesn't warn the user if it encounters such KML. The following is taken from the release notes for Google Earth 6.1:

“To conform more closely with the OGC KML 2.2 standard, we've made the parsing of the <coordinates> tag more strict in version 6.1. Spaces must only be used to separate the (longitude,latitude,altitude) triplets which specify the coordinates of a single point. It is invalid to add spaces between individual coordinates. We understand that previous versions of the client were more lenient, so if your KML files contain spaces between coordinates in coordinate triplets, an easy workaround is to remove all whitespace located immediately before or after commas inside a <coordinates> tag.”

Bad KML may render correctly in Google Earth but may fail to load or view correctly in some of the many applications that import KML and KMZ files. Developers of such applications must implement software to the strict KML standard because that is what is documented. Developers do not have the resources or the time to test every possible case, so their environment requires more rigid interfaces, which in turn demand more adherence to specifications.

As soon as a given KML file is accessible via a URL or is emailed elsewhere, its use is uncertain. In such cases, Google Earth may not be the only consumer, and additional validation tools are necessary. It is likely that large systems making data available in KML will be consumed by multiple applications unknown to the data provider. For time-critical data supporting natural disasters, national security, homeland security, airline safety, etc., using the standard correctly is essential. During disasters such as the 2011 Tohoku earthquake or Hurricane Katrina is not the time to learn that KML data is not correct and that critical decisions have been made based on incorrect geo-referenced data.

The benefits of data standards such as KML can be achieved by incorporating validation in the design and deployment of KML data and services.

Types of Validation

There are three classes of KML validation that apply to XML-based standards.

1. Well-formed validation. This validates a given XML document is well-formed with respect to XML standard meaning:

  • The XML document has a single root element.
  • All start and end tags are matched and names are case sensitive.
  • Attribute names in elements are unique (i.e., no duplicate names).
  • Content contains only characters allowed by the appropriate encoding (e.g., UTF-8, etc.).
In most cases, a KML document will not load in Google Earth™ if it is not well formed.

A common error found in KML files involves labeling the file with the default UTF-8 encoding in the XML prolog while inadvertently copying/pasting an international character or graphic symbol (e.g., the degrees symbol ) from an email or document in the description field of one of the placemarks. Google Earth ignores such errors, but strict XML parsers fail to parse them and abort loading these documents with a fatal parse error. Replacing the UTF-8 encoding with ISO-8859-1 resolves this issue.

2. Schema validation. Valid XML documents conform to declared XML Schema specifications and its structure is valid as defined by the Schema.

  • Schema defines a strict order of elements and spelling of element/attribute names which are case-sensitive.
  • KML schema defines the hierarchy of abstract and concrete elements and as such what elements/attributes are inherited.
  • KML schema defines where non-KML namespaces may appear.
In case of KML, valid documents conform to the published OGC KML 2.2 XML Schema and additional schemas as defined in the given KML document instance.

3. Specification validation. Valid KML documents conform not only to the KML XML Schema but also follow the strict design and business rules as defined in the specification. Many such rules are not covered by KML XML Schema validation.

For example: KML specification defines the following rules:

  • LineString geometries consist of two or more coordinate tuples, each consisting of floating point values for longitude, latitude, and altitude. The altitude component is optional. Space must be inserted between tuples. Do not include spaces within a tuple.

  • LinearRing contains four or more tuples, each consisting of floating point values for longitude, latitude, and altitude. The altitude component is optional. Do not include spaces within a tuple. The last coordinate must be the same as the first coordinate. Coordinates are expressed in decimal degrees only.
The coordinates component of a LineString and LinearRing from the KML XML Schema perspective must only be a list of strings separated by whitespace so "a b c" is valid by its schema type definition and likewise there is no range checking of latitude/longitude values within coordinate tuples.
  • The value of the following Feature elements shall be inherited by all Feature members of a hierarchy: atom:author, atom:link, Region, and TimePrimitive, unless overruled by the presence of such elements locally. Inheritance of these elements continues to any depth of nesting, but if overruled by a local declaration, then the new value is inherited by all its children in turn.

Additional discussion of validation can be found in the KML Errata.

KML Validation Tools

Basic Validation - Well-formed XML test

The quickest way to validate a KML file is using your web browser. KML is an XML file so first you can test if it's a well-formed XML file, which is a prerequisite to it being a valid KML file.

Most web browsers validate XML files you can simply rename KML file with an .xml file extension then drag the file onto a web browser (Firefox, Chrome, etc.). The web browser will check if the file is well-formed. If not well-formed (e.g. mismatched tag, missing closing tag, etc.) it will show the line and column numbers were the KML file fails validation. If a KML file does not generate an error it means it is well-formed which may or may be a valid KML file.

If the KML fails the well-formed test then you have to fix these errors before addressing the KML specification-related errors.

Example:

Here's a KML file that has a common XML error. Such errors can happen when copying KML snippets from one file to another without copying the relevant references.

1: <?xml version="1.0" encoding="UTF-8"?>
2: <kml xmlns="http://www.opengis.net/kml/2.2">
3:    <Placemark>
4:      <name>Cairo, Egypt</name>
5:      <atom:link href="https://maps.google.com/?ll=30.098207,31.246104..." />
6:      <description>The Great Pyramid of Giza</description>
7:      <Point>
8:        <coordinates>31.133866,29.979227</coordinates>
9:      </Point>
10:    </Placemark>
11: </kml>
If you rename the file badAtom.kml to badAtom.kml.xml then drag the file onto a web browser, you would see the error highlighted like this where the line number and column number show the context of the error.

XML Parsing Error: prefix not bound to a namespace
Location: file:///C:/temp/badAtom.kml.xml
Line Number 5, Column 5:
    <atom:link href="https://maps.google.com/?ll=37.50002,127.03656..." />
----^

In cases where you're not sure how to fix the KML you can just add an XML comment that wraps the bad line(s) to skip it for now until the file validates without any errors. Here's what a comment around the bad line would look like:

1: <?xml version="1.0" encoding="UTF-8"?>
2: <kml xmlns="http://www.opengis.net/kml/2.2">
3:    <Placemark>
4:      <name>Cairo, Egypt</name>
5:      <!--
6:        <atom:link href="https://maps.google.com/?ll=30.098207,31.246104..." />
7:      -->
8:      <description>The Great Pyramid of Giza</description>
9:      <Point>
10:        <coordinates>31.133866,29.979227</coordinates>
11:      </Point>
12:  </Placemark>
13: </kml>
Now running this KML file through an XML validator would show it as valid (assuming no other errors are present). This technique allows you to quickly isolate where the KML is broken and either fix it immediately or suppress the error to fix later.

Here's the valid KML file with correction highlighted:

1: <?xml version="1.0" encoding="UTF-8"?>
2: <kml xmlns="http://www.opengis.net/kml/2.2" xmlns:atom="http://www.w3.org/2005/Atom">
3:    <Placemark>
4:      <name>Cairo, Egypt</name>
5:      <atom:link href="https://maps.google.com/?ll=30.098207,31.246104..." />
6:      <description>The Great Pyramid of Giza</description>
7:      <Point>
8:        <coordinates>31.133866,29.979227</coordinates>
9:      </Point>
10:    </Placemark>
11: </kml>
Additional tips to manage your saved places file and repair broken KML can be found below.

KML Schema Validation tools

There are some web sites that validate KML by URL or file upload: Note that the Galdos Systems KML validator above does not validate KML extensions (i.e. gx namespace) so for example this valid KML sample shows 2 errors related to gx: namespace. Also, the KML validator does not support https protocol so you must either make sure there is a http: URL to the target KML file or first download the KML file locally then upload the file to validate it.

There is a standalone command-line tool written in Java, XmlValidate that can validate any XML file or URL including KML and KMZ files including those with KML extensions against its target XML Schemas. This tool allows bulk validation given a target directory in addition to forcing a target XML Schema other than the one defined in declared namespace (e.g. validate KML 2.2 documents against KML 2.1 schema). Note this tool validates KML/KMZ files to the relevant KML XML Schema and not the stricter specification rules as defined in the standard so for such conformance testing you need to use a service like the Galdos Systems KML validator.


Managing Saved Places KML in Google Earth

Google Earth™ manages the saved places just as web browsers manage their bookmarks or favorites. In the case of Google Earth, it saves the "favorites" in a single KML file named myplaces.kml. There are a number of misperceptions about the saved places file as well as how to properly manage it. If not managed then it's likely it will get too large to manage and slow down Google Earth when it tries to load and save changes. This can also cause Google Earth to crash, which in turn could corrupt the saved places file if it crashes while saving the file.

Maintenance and periodic cleaning of your saved places

Unlike a bookmark or favorite in a web browser which is just a URL and last access date, most KML you open gets saved in its entirety to your saved places (often many megabytes in size) rather than just a URL to the resource on the Internet. Well-behaved KML feeds (e.g. USGS Earthquake KML feed) on the other hand do just that and use a small KML document with NetworkLink to fetch the KML contents from the network to your Google Earth client and only the small KML document is saved locally.

The saved places file grows over time if not properly maintained with periodic cleaning

To make matters worse, KMZ files when opened in Google Earth are expanded into its raw KML (sometimes 10x or 100x original size) and appended to the saved places file (if saved). Over time the saved places file gets larger and larger often over over 100MB. The myplaces.kml is an XML file that must be parsed every time you start Google Earth (and loaded into memory) often adding to the initial delay when Google Earth starts and a delay every time the saved places are written out to disk. It's often reported by users in the Google Earth forum that the saved places file got corrupted and repairing a large XML file is cumbersome and time consuming.

Best practice is to keep low memory footprint with small saved places file typically less than 10MB

Once a month or at least once a year it is best to review your saves places and archive your old/unused KML features.
  1. Create a new sub-folder on your computer to store your KML and KMZ files (e.g., under 'My Documents' - create a KML folder).
  2. Using Google Earth you can select a folder of KML features (e.g,. Placemarks, Image Overlays, etc.) then save it to a local KMZ file in this new directory after which you can delete the features from your saved places. Recommend to save large collections as "KMZ" files to conserve disk space.
  3. Now you can create a NetworkLink in Google Earth to auto-load this KML or KMZ file when you start Google Earth or simply load your archived files as needed as you would your separate documents in your 'My Documents' folder. Also add the <visibility>0</visibility> element to most of your NetworkLinks to make them load only when you check them manually. Loading all the places at once may take up too much memory and slow down the initial load of Google Earth.

Best Practice for KML Developers

Best practice for KML authors is to provide a NetworkLink as the KML launch point to Google Earth to load the main KML content preferably stored in a separate KMZ file if large to minimize download times so users of it don't have the entire KML saved locally. A benefit for this is when the main content changes on the server the end-users get the latest and greatest since they only access it via a network reference and never store the main KML content permanently. If, for example, you examine the USGS Earthquake Network Link feeds you'll find a KML snippet such as the following.
	<NetworkLink>
		<name>Colored by Age</name>
		<visibility>1</visibility>
		<open>0</open>
		<Snippet maxLines="1">Updates every 5 minutes</Snippet>
		<Link>
			<href>http://earthquake.usgs.gov/earthquakes/feed/kmlage/1.0/week</href>
			<refreshMode>onInterval</refreshMode>
			<refreshInterval>300</refreshInterval>
		</Link>
	</NetworkLink>
Notice the refreshMode and refreshInterval tags which specify that the KML auto-refreshes every 5 minutes. The bulk of the data are loaded from the target URLs (e.g., http://earthquake.usgs.gov/earthquakes/feed/kmlage/1.0/week) that are specified in several NetworkLinks in the root KML file.

Repairing a damaged or corrupt KML file

Some tips to repair a corrupt KML file can be found in this document. One such trick is to rename the KML file to have an .xml extension and use a web browser to show the line/column where an error exists.

A lengthy discussion to repair corrupt KML can be found here in the Google Earth Help Forum.

Backups

A common misconception is that the saved places in Google Earth are located in the "cloud" such as your GMail or Google docs - sadly it is not (at least not yet). The "sign in" option in Google Earth further makes it appear that the saved places will be stored along with your Google mail, docs, etc. and logging into another computer and launching Google Earth would automatically show the same saved places, which is not the case. The saved places are stored locally in a myplaces.kml file on your computer so you must make regular backups of this file (and other critical documents) to avoid losing your data.
The location of this file depends on the version of the operating system and Google provides a support article to locate the myplaces.kml file on your computer.
Retrieving and moving your placemarks file

Google Earth when started first makes a copy of saved places file myplaces.kml to myplaces.backup.kml and a working copy as myplaces.kml.tmp. If, however, the saved places file gets corrupt and the user stop and restarts Google Earth to "fix" it, the process of restarting Google Earth would replace the last working backup with the corrupt copy -- not what you want to do.

Here's a script I wrote to maintain the last 31 days of saved places created by Google Earth:

https://productforums.google.com/d/msg/earth/RIpSj7vBeWo/C3bB-F1u6woJ
I use a variation of this technique to keep multiple copies of my saved places file in addition to periodic cleaning and archiving of old or unused KML features.

KML Attribution / Contribution

Organizations should use a disciplined approach in how they deliver their KML services. Without such an approach, information within the KML files can become outdated or used inappropriately, resulting in decisions based on misleading or obsolete information. Attribution such as the following should be included within the KML data to ensure that user expectations are clearly defined with regard to the data provided via KML.

  • Information Current As Of Date
  • Frequency of KML file/service updates
  • Office of Primary Responsibility
  • Contact Phone Number / Website / Email Address
  • Source of Information
  • ...
Note that the exact set of metadata fields depends on the particular information domain.
The metadata can be added using the following three methods:
  1. by adding the data as XML comments in the KML file,
  2. by adding the text to the description of the first KML feature, and/or
  3. by adding the data as ExtendedData defined in the outermost/first KML Feature.
These methods can be combined; they are not mutually exclusive. The recommended best practice is to use the second and third methods where the metadata is viewable by the end-user.

Add Attribution as XML comments in the KML file

The simplest but least desirable method is to include the metadata as unformatted text nested inside an XML comment at the start of the KML file as shown below. The attribution metadata is included with minimal change to the KML, but the data is hidden from the casual user. To view this text, an end user would need to view the source of the KML as XML. Typically only a developer would attempt to view the raw KML data. This is the quickest approach to just annotate the KML with appropriate attribution text.

<?xml version="1.0" encoding="UTF-8"?>
<!--
Information Current As Of 2010-01-25T08:00:43Z
Frequency of KML file/service updates:
	KML updated daily at 0800Z or something like NetworkLink auto-refreshes with default update rate at 60 second interval.
	Data updates are available every 10 seconds so clients can change the update interval to no less than 10 seconds.
Office of Primary Responsibility: My Company
Contact Phone Number / Website / Email Address: 123-555-1212 anyuser@domain.any.org
Source of Information: XYZ System
...
-->
<kml xmlns="http://www.opengis.net/kml/2.2">
<Document>
...
</Document>
</kml>

Add Attribution to description

The second method is to add metadata attribution to the description element of the outermost container element in the KML. This allows producers to add attribution metadata to a KML Feature, where the description is displayed in a pop-up balloon in Google Earth™ when an end user clicks on that feature. Producers can format the description using the richness of HTML, including JavaScript and CSS. Consider adding a snippet with a short description that appears in the menu list view while the full description appears in the Feature's pop-up balloon.

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2">
<Document>
  <Snippet>
    Short description of the data/service
  </Snippet>
  <description>
<![CDATA[
Description of the data/service
 
<B>Information Current As Of 2010-01-25T08:00:43Z</B><BR>
<B>Frequency of KML file/service updates:</B><BR>
 NetworkLink auto-refreshes with default update rate at 60 second interval.<BR>
 Data updates are available every 10 seconds so clients can change the update interval to no less than 10 seconds.<BR>
<B>Office of Primary Responsibility:</B> My Company<BR>
<B>Contact Phone Number / Website / Email Address:</B><BR>
 123-555-1212 anyuser@domain.any.org<BR>
<B>Source of Information:</B> XYZ System<BR>
...
  </description>
...
</Document>
</kml>

Add Attribution as ExtendedData

The third approach involves adding the metadata in an ExtendedData element to the KML document. This method is a machine-parsable implementation; the other methods added metadata as unstructured free-form text. Example below shows the raw KML text for this approach. As in the second approach, adding attribution to the description, in this method the end user can view the metadata formatted in a pop-up dialog box when it is viewed in Google Earth™. If the Document element has no description tag then the ExtendedData is autoformatted in Google Earth™ as an HTML table with a border around the table cells. The ExtendedData values also can be inserted into a custom description or BalloonStyle template using Entity Replacement for Extended Data Elements, with the names of the ExtendedData elements as placeholders (e.g. $[ContactPhone]).

For further details see Google's Tutorial on Adding Custom Data to KML.

<?xml version="1.0" encoding="UTF-8"?>
<kml xmlns="http://www.opengis.net/kml/2.2">
<Document>
      <ExtendedData>
        <Data name="Information Current As Of">
          <value>2010-01-25T08:00:43Z</value>
        </Data>
        <Data name="Update Frequency">
          <value>NetworkLink auto-refreshes with default update rate at 60 second interval.
 Data updates are available every 10 seconds so clients can change the update interval to no less than 10 seconds.</value>
        </Data>
        <Data name="Office of Primary Responsibility">
          <value>Data Owner</value>
        </Data>
        <Data name="ContactPhone">
          <value>123-555-1212</value>
        </Data>
        <Data name="ContactEmail">
          <value>anyuser@domain.any.org</value>
        </Data>
        <Data name="Source">
          <value>XYZ System</value>
        </Data>        
      </ExtendedData>
...
</Document>
</kml>

Reference

Google Earth (TM) and Google Maps (TM) are registered trademarks of Google Inc. KML4Earth is not affiliated with Google.
KML4Earth is not an official source of information on Google or Google Earth. But, we try to get reliable information whenever we can.

Created and maintained by jasonm1