Yahoo's weather IPA in Free Pascal

In February, I looked at how to recover weather conditions and forecasts from Yahoo Weather with a program written in Pascal. After the recent installation of Free Pascal / Lazarus on Ubuntu 18.04, I decided to extend the study to other APIs: Apixu, Dark Sky, MET Norway, OpenWeatherMap and Weatherbit.io.

Table of content

1. REST Architecture
2. Language Support
3. Specification of the Location
4. Weather forecast and Historical Data
5. Data Update Intervals
6. Limits
7. Demonstration Programs

REST Architecture

Formally, these weather services are more or less compliant with the REST architecture: a resource, such as weather conditions at a specific location, is obtained from a Web server with a properly formatted HTTP or HTTPS request. In principle, the protocol must be stateless: server and the client should not keep any information. Thus, each request is independent, not related to the previous one or the next one. It is therefore necessary that each request from the client contains all the information necessary to obtain the desired resource and that the response of the server contains all the required information.

In practice, sites keep information about each request. With two exceptions, HTTP/HTTPS requests made to a server must contain a "key" that is obtained by registering with the provider. They can then track usage for pricing purposes. Usage is limited and the provider hopes to sell subscriptions to better plans later on. Presumably, Yahoo Weather and MET Norway, where it is not necessary to register to get a key, use the client's IP address to control the number of requests made.

Here is the request that must be sent to obtain the current conditions in Paris, Ontario. This Canadian community of just over 12,000 inhabitants is one of 17 communities of the same name listed by the Yahoo geolocation service.

The last five examples have the classic form of a URL with a query string joined by the "?" separator to the base address. The query consists of one or more name=valeur couples separated with a "&". The first example follows this pattern, but the value of the q (probably an abbreviation for query) field is actually an expression that complies with Yahoo's structured query languag, YQL, whose syntax is similar to SQL.

All services accept requests made using secure HTTP (HTTPS). Dark Sky denies queries that are not secure, Yahoo Weather and MET Norway redirect them to secure protocol and OpenWeatherMap, Weatherbit.io and Apixu accept unsecured HTTP requests.

Almost all servers return weather data formatted in JSON or XML but the default format is different depending on the service. The exception is MET Norway which only uses XML and Apixu where the desired format must be specified with a different web address instead of a supplementary query parameter.

The following table summarizes this information about accepted protocols for queries, the need to register for a key, and the formats available for the results. The links in the table are to the registration pages to get a key. It's free and there is no need to provide information other than a valid email address.

(*) denotes the default format of the response when a format is not
specified in the query and there is more than one possibility.

Obviously, each service presents the data in its own way, it would be far too simple for them to adopt a uniform JSON template. Moreover, the current weather data returned by the services are not the same. However, some observations are universally available including temperature, humidity, and atmospheric pressure. On the other hand, cloudiness and precipitation, if any, are not available from all services.

Language Support

All services give a short textual description of the weather conditions. It's often just one word such as "cloudy" or "storms", but the text can also be longer. Almost all servers use English as the default language for these descriptions. However, other language can be specified, but the list of supported languages ​​differs from one provider to another. The main European languages, French, Spanish, German, etc. are all available. The exceptions are Yahoo Weather and MET Norway which use, respectively, English and Norwegian only.

Where applicable, the language is specified with a query parameter. Typically the form is lang=xx where xx is a two-letter language code, with an a additional dialect identification suffix if needed.

There is duplication like Ukrainian. OpenWeatherMap denotes it with the code "ua" while other providers use "uk". It seems to me that the latter are right, because "uk" is the ISO 639-1 code for the language whereas "ua" is the ISO 3166-1 alpha 2 code for Ukraine where other languages than Ukrainian are spoken.

All services include a numeric code with the textual description of current conditions. Obviously, the codes are different depending on the provider. The list of codes can be found on their website and it is therefore possible to translate the descriptive text if it is not available in the desired language.

Specification of the Location

Some services offer more than one way of identifying the location for which meteorological measurements are desired. In all cases, the coordinates of the place (latitude and longitude) can be used. Moreover, it is the only method accepted by Dark Sky and MET Norway.

Other providers (Apixu, OpenWeatherMap, Weatherbit.io and Yahoo Weather) accept the name of a locality. It may be natural and practical to use this way of specifying a place, but there may be several places with the same name. Born in Montréal, Québec, I found it amusing to visit Montréal in the Aude region of France a few years ago. In fact, there is an Association des Montréal de France et d'Europe which brings together 7 communities, including 6 in France. Yahoo's WOEID database (Where on Earth Identifier) associates "Montréal" with 9 places and "Montreal" with 27. To circumvent this common problem, the suppliers add one or two additional fields to specify the country and, possibly, an administrative region.

There are other, more specific ways to specify the location such as the postal code or the code of the nearest weather station or airport. Some services have a list of localities with a unique identifier for each. This avoids any ambiguity as long as the location is in the provider's list.

The OpenWeather and Weatherbit.io location identifier databases can be downloaded. Just follow the link in the table above. The WOEID database was publicly available from 2009 to 2012. As far as I know, it can no longer be downloaded, but it is accessed with YQL queries to the Yahoo IPA.

In principle, latitude and longitude coordinates, IATA or ICAO airport codes and weather station identifiers do not cause any difficulties. The location identifier of OpenWeather, Weatherbit.io and Yahoo Weather should be as accurate. However this is not exactly the case. If you look at a part of the OpenWeatherMap (city.list.json) database for "Montréal" and "Montreal" in France, there are several places differentiated only by their geographical coordinates.

It will be necessary to find elsewhere the geographic coordinates of all the Montréal in France to determine if the region is Aude (Occitanie), Ardèche (Rhônes-Alpes), Burgundy and so on. This is more or less the same problem with the name Weatherbit.io database cities_all.csv.

Unfortunately, the states.csv database gives no information as to what is state_code 76.

Querying the Weateherbit.io API with the city identifiers will return their geographic coordinates.

Using maps, atlases and so on, I found that the first is Montreal-du-Gers and the second is Montreal in Aude, both in the Occitanie region. Given this database we are not surprised to learn that specifying Aude, Languedoc-Roussillon or Occitanie as a region does not help Weatherbit.io to find Montreal in Aude. Yahoo is more comprehensive providing up to two administrative regions in addition to the country.

The WOID database does not take into account the territorial reform of 2016 that merged Languedoc-Roussillon and Midi-Pyrénées to form Occitanie.

Clearly, nothing is perfect. And this finding extends to the postal code. Apixu reported that "75001, US" was Addison, Texas which is correct according to the US Postal Service. According to Weatherbit.io, the same code represents Dallas, TX and according to OpenWeatherMap this zip code identifies Plano, TX. Plano and Addison are in the outer suburbs of Dallas.

After that, it may be surprising, but "75001, FR" is the French capital according to OpenWeatherMap, Weatherbit.io and Yahoo. However, OpenWeatherMap can not decode UK postcodes and can only interpret the first few Canadian postal codes in a few days. Apixu does not know the French postal codes, but, on the other hand, it seems to process the postal codes of the United States, the United Kingdom and Canada without problem.

Weather Forecasts and Historical Data

In addition to current conditions, forecasts can be obtained. Either a different URL or a change to the query element is required to get these predictions. The exception is, again, MET Norway, which gives observations and forecasts together. Let's also mention that Dark Sky runs backwards; query parameters are used to exclude elements from the response, otherwise, all the available forecasts are returned along with the current conditions and all the available forecasts together.

The following table summarizes the free forecasts available from the supplier. Historical data can also be obtained. This question interests me much less so do not rely too much on this column that only repeats what is found in the online documentation that are is not necessarily up to date.

Suppliers significantly limit the free historical data available. Typically, only a single day's data from the previous 30 days can be obtained.

Data Update Intervals

With the exception of Met Norway, the services examined here do not produce meteorological data. They get their data from various sources and then synthesize it for presentation to their customers. As a result, the interval between updates of current conditions and forecasts depends in part on the time between updates of meteorological data the providers use, and in part on the provider who integrates the data according to its chosen pace. Consequently, the information about the time intervals between updates is not very clear.

I did not find anything about this subject with respect to Apixu, MET Norway and Yahoo Weather. For others here is what I gleaned.

Dark Sky

How often is your data updated? "Next-hour minutely forecast data is updated every five minutes. Hourly and daily forecast data are updated every hour. Severe weather alerts are updated in real-time."

OpenWeatherMap

Data is not accurate "The update interval can be up to a couple of hours for free/start-up API servers". Look at the Price. page also.

Weatherbit.io

FAQ "How frequently do does the current weather API update? Data updates as fast as it is reported. This can vary by location. Typically between 15-30 mins. How frequently do forecasts update?Forecasts update once per hour in most locations."

Limits

The Norwegian Meteorological Institute (MET Norway) is a public company whose

Other providers are for-profit companies. What drives them to give free access to some of their services is not immediately obvious. Is it seen as advertising or a loss leader? No matter, the future of free services is not guaranteed. Moreover, private providers have stopped offering free access and others have never adopted this tactic.

All providers reviewed here limit the number of requests accepted in a given interval. If the limits are exceeded, then non-free package must be purchases. The table below shows the limits for free use. As mentioned at the beginning even if a service is free, it is nevertheless necessary to register to obtain a key that must be included in any request except for MET Norway and Yahoo Weather.

Demonstration Programs

The archive weather_api_src.zip contains the source code for the six programs that access each provider's IPA. There is also a program for querying the Yahoo WOEID database.

The programs were developed with Free Pascal (version 3.1.1) Lazarus (version 1.9.0) (revision SVN 58721) on the Ubuntu platform 18.04.1. The DCPcrypt package needs to be installed to encrypt and decrypt the keys. It is found in the Lazarus Code and Component Repository (CCR). This can be done with the fpcpudeluxe installer. See my ticket Installing Free Pascal and Lazarus with fpcupdeluxe on Ubuntu 18.0.4. The error messages the latter displays are of no consequence.

The initial demonstration program for Yahoo Weather is still available. The latter used the Ararat Synapse package to access the Web. Instead, the new programs use fphttpclient included in standard Free Pascal library. So there is nothing to install. If one prefers to continue using Synapse, just put httprequest_synapse in place of httprequest the uses statement of the implementation section of the source file common/getweather.pas.

The first time a demonstration program is used, the weather IPA key must be entered (if needed of course). Click the Key tab, enter the key in the text field, and then click the Set button. As soon as the button is activated, the contents of the key field are deleted. Thus, the key is never displayed except when entering its value should visualization of the key been activated.

The key is saved in an encrypted file in the directory containing the executable program. From then on, the key will be read from this file. The security phrase used to encrypt the key is obviously the same for all copies of the software if you do not change anything. It is possible to specify this phrase every time the application starts. The following message is displayed at the beginning with instructions on this subject.

This is my first foray into the world of cryptography. So I do not have much confidence in the security of the keys stored with these programs.

Here is a screen shot of the demonstration program owm_demo; the others are similar.

The method to specify the location is chosen with the tab along the top. After, just fill in the fields displayed under the tab, select the desired meteorological data from the drop-down list Fetch. One can also choose the language used for the descriptions of the conditions, the units of measure, the format of the answer and the protocol to be used.

To finally send the request and obtain a response from the provider, click on the Raw or Structured button. In the first case, we obtain the raw response of the supplier. In the second case, the response is analyzed and displayed in a tree. Its contextual menu could be useful.

Clicking the URL button updates the query based on chosen parameters and displays it in the scroll box at the centre of the application except that the key is not displayed; its position is indicated by {key}. The request and the answer can be copied to the clipboard.

Yahoo's weather IPA in Free Pascal