| How To |
IDMP stations - Data quality control - How to join the Network - Data exchange These pages describe what is called an IDMP station, how the quality of IDMP data is insured, how to join the IDMP network and publish the documentation of your station, and how to get the data from IDMP stations. We welcome our first sponsor: the EKO Instruments Trading CO., Ltd, in Japan (http://eko.co.jp). Manufacturers are invited to help us in maintaining the server. If you are interested, contact the IDMP WebMaster, he will set up a link to your web server, if you have one, or if it is not the case, he will build a few pages for you.
What is an IDMP station ? What you will find below is a short summary of what is meant by IDMP station. After reading this section, if you are interested in setting up a station, you should first get a copy of the «Guide to recommended practice of daylight measurement» published by the CIE under the code 108:1994. For this, contact the CIE representatives in your country. In the following, this publication will be simply called the IDMP guide. It should be clear to the reader that the CIE does not provide any fund to set up an IDMP station, therefore you will have to look for private or public funds in your country. The international framework created by the CIE, the activities of TC 3.25, this web server, should give you some help. If you need technical assistance, you may also contact members of the IDMP network. The site on which to set up an IDMP station should be selected so that continuous obstructions to light are not higher than 4.6 degrees. It is an absolute requirement that an operator be available daily to take care of the station and its data. There are three classes of IDMP stations based on the number of measurements they perform: basic, general and research. The cost of the station will of course increase with the number of measurements and the type of sensors. All IDMP stations perform measurements at a frequency of at least one minute. The IDMP guide defines a glossary of terms for the measurements taken by the IDMP stations. These terms are indicated below between parentheses, you should stick to them. For information on sensors, manufacturers are setting up web servers which contain a complete description of their products, see for instance the «Meteorological Instruments» pages of EKO, our first sponsor. The Basic class IDMP stations record at least: (1) the global horizontal illuminance (Evg) (2) the diffuse horizontal illuminance (Evd) (3) the global horizontal irradiance (Eeg) (4) the diffuse horizontal irradiance (Eed) The diffuse component is measured using a ring or a disk to shade the sensor from direct sunlight. It is recommended to use a disk which causes less obstruction than the standard shade ring. The IDMP guide provides information on the size of rings, and on the obstruction correction factor, to apply to the diffuse component. The shading systems used for the irradiance and the illuminance sensors should be identical so that the correction factor do not interfere in the computation of luminous efficacies. The measurements of the diffuse components may be replaced by the measurements of the direct normal components: the direct normal illuminance (Evs) and the direct normal irradiance (Ees). The IDMP guide provides recommendations on the field of view of the direct normal sensors. Since measuring the direct normal component requires to follow the sun path, the apparatus required will be more expensive. The General class IDMP stations record at least: (1) the global horizontal illuminance (Evg) (2) the diffuse horizontal illuminance (Evd) (3) the global vertical illuminance in the four cardinal directions: North, East, South and West (Evgn, Evge, Evgs, Evgw) (4) the global horizontal irradiance (Eeg) (5) the diffuse horizontal irradiance (Eed) (6) the zenith luminance (Lvz) The sensors measuring illuminance on vertical surfaces are screened from the ground reflected light by a matt black screen forming an artificial horizon. The zenith luminance sensor has a field of view defined by an half open angle of 5.5 degrees. The Research class IDMP stations record at least: (1) the global horizontal illuminance (Evg) (2) the diffuse horizontal illuminance (Evd) (3) the direct normal illuminance (Evs) (4) the global vertical illuminance in the four cardinal directions: North, East, South and West (Evgn, Evge, Evgs, Evgw) (5) the global horizontal irradiance (Eeg) (6) the diffuse horizontal irradiance (Eed) (7) the direct normal irradiance (Ees) (8) the global vertical irradiance in the four cardinal directions: North, East, South and West (Eegn, Eege, Eegs, Eegw) (9) the air temperature (Dbt) (10) the relative humidity (Rh) or the dew point temperature (Det) (11) the zenith luminance (Lvz) (12) the sky luminance distribution (Lv) The measurement of the sky luminance distribution is an important feature of a research class station. Sky luminance measurements are performed at least every 30 minutes, centered at 15 and 45 minutes after the hour, according to a pattern of 145 points described in the IDMP guide. The whole pattern is covered in one minute. Sky scanners are available from a few manufacturers. Some research laboratories such as the University of Technology in Eindhoven (IDMP station NLD1) have built their own. In addition to the measurements indicated above, sunshine duration (S), cloud amount (N), Wind speed (Ws) and direction (Wd) at 10 meters, can be measured at the station or obtained from a nearby weather service. The sunshine duration is recorded as a burnt trace on a standardized card of the WMO International Reference Sunshine Recorder, or from the time during which the direct normal irradiance (Ees) is at least 120 W/m2. The total and directional cloud amount is based on a subdivision of the sky into 19 zones. It is assessed using all sky photographs, video based images in two filters (blue-red) or on site observations. Again, the IDMP guide provides detailed guidelines on the assessment of cloud cover. At the moment, only two IDMP stations (Sheffield/GBR4 and Ann Arbor/USA1) provide directly the cloud cover information recommended by the IDMP guide. This is due to the fact that there is not yet a perfect method to compute automatically the cloud cover from sky images, and also because images add a lot of weight to data storage. Other IDMP stations (Vaulx-en-Velin/FRA2) get hourly cloud cover and cloud type indirectly from observations made at a nearby weather station. However, the development of digital cameras which can be equipped with fish eye lenses (such as the Nikon F2) as well as the development of new image compression schemes, should allow to add full sky images, to the database of each IDMP station. The cloud cover and the sky luminance distribution could be computed from these images. All the sensors used in the IDMP stations have the accuracy recommended in CIE publications. CIE publication 69 describes the evaluation of photometers. The WMO «Guide to Meteorological Instruments and Methods of Observation» describes the evaluation of radiometers. The IDMP guide provides additional recommendations. CIE TC 3.25 has organized an intercomparison of the sensors used in all IDMP stations by circulating reference photometers (see the current state of this comparison in the CIE TC 3.25 section).
Data quality control Before being processed into end-use products that may be exchanged among IDMP participants and other interested parties, each measurement performed by an IDMP station is subjected to rigorous automatic and manual quality control. Automatic data quality control allows the early detection of sensor problems. It consists of absolute data checks to insure that data fall within expected bounds, as well as coherence tests to compare the outputs of two or more sensors. Common problems include sun tracking misalignment, sensor soiling, large calibration drifts or other obvious malfunctions. It is imperative to perform automatic data quality control as early after acquisition as possible, or even in real time, if permitted by the acquisition system. The IDMP guide describes two automatic data quality control procedures: (1) One has been implemented and evaluated since 1991 at several IDMP stations, it is based on fixed acceptance thresholds. Evaluation has shown that it may not be 100% effective in the case of small errors, such as small sun-tracking errors or small shadow band misalignment. The source code (in Fortran) developed initially by H. Porter from the Building Research Establishment (IDMP station: GBR1) for a JOULE European Research Programme, then adapted by ENTPE for its station (IDMP station: FRA2) is available for download, from the Library section. (2) A more effective automatic data quality control has been developed by Ineichen et al., at the Group of Applied Physics in Geneva (IDMP station: CHE1). It is fully described in Appendix 1 of the IDMP guide. Its strength resides in the use of thoroughly validated models that modulate narrower data acceptance thresholds as a function of insolation conditions and solar geometry, thus increasing measurement error detection effectiveness. The whole procedure has been coded in a stand alone program distributed with the IDMP guide. This program, called AQCCIE, works on 386 machines and higher, under the MS-DOS or Windows operating systems. It is fully compatible with the data format recommended by the CIE. Automatic quality control is a necessary but not sufficient component of quality control. Manual quality control is necessary to complement overall quality control. The IDMP guide provides recommendations on instrument and data inspection. It strongly suggests the systematic use of «maintenance sheets» and «data plots». In the Library section, you can download an example of a maintenance sheet and a demonstration version of a program that can be used to plot the IDMP data. This program called TIMElux has been developed by ENTPE (IDMP station: FRA2). It uses a calendar interface to plot the data measured by an IDMP station in an entire month, on one page. The presentation is attractive and looks good on screen as well as on paper. It is available for the MS-DOS, Windows and Macintosh operating systems. It is fully compatible with the data format recommended by the CIE.
How to join the IDMP network ? You become part of the IDMP network, once you have demonstrated that you have set up a measuring station, which strictly follows the recommendations summarized above and fully described in the IDMP guide. For this, prepare a document describing your station, using a format similar to the one used for the IDMP stations presented on this server (a description of the format is given below). If you have not done it yet, contact the chairman of TC 3.25: Prof. Hiroshi Nakamura or one of the two secretaries: Dr. Yasuko Koga and Dr. Dominique Dumortier, who will validate your station based on this document. A visit to your station may also be arranged. You will also have to engage to exchange at least one year of data for free, with other members of the network. Once you join the IDMP network, you become a member of Technical Committee 3.25. A code will be given to your station. It consists of your country code (three letter according to ISO 3166:1993) followed by a number attributed by chronological order to each station within the same country. For instance, the ISO three letter code for India is IND, the station in Roorkee was the first one in India, so its code is IND1. The format used to describe an IDMP station is strictly defined. It consists of five sections: Contacts, Site, Station, Views and Maintenance. If you want to show more than what the format allows: more pictures, QuickTime VR tours of the station, live recorded data... Create your own server and ask the IDMP WebMaster to set up a link to this server, in the Contacts section. If you wish, the WebMaster will also customize the «IDMPNetwork» banner to include the name of your station. Pictures and maps required to produce the description should be sent by regular mail, the IDMP WebMaster will digitize them for you. All the originals will be sent back to you if you request it. Text can be sent via electronic mail or regular mail, as long as it is on a disk. The Contacts section contains general information on the station: who manages it ?, where is it ?. A topographic map shows the land within a 5 km radius of the station location. Do not forget your country code in the phone and fax information field, indicate your electronic mail address and the URL of your web server, if you have one. For latitude and longitude, mention North or South of the Equator and East or West of Greenwich. There is no need for you to provide a map of your country, we have them in our database. The class of the station should be any of the following three words: Basic, General or Research. The operation date is the date at which you can provide reliable data. If you are still testing the station, then you should mention it. For the topographic map, provide an original map so that we can get the best results when reproducing it. Use only a pencil to write something on the map. We will put the label «IDMP Station» ourselves. Indicate the scale which should be as close as possible to 1:70,000. If it is a bigger scale, we will make it fit into the space reserved. Indicate where your station is located and where the closest weather station is (if it is located within a 5 km radius). Specify the North orientation. The Site section gives a description of the ground surface within a 5 km radius as well as the description of the climate and any local factors such as pollution sources which may affect daylight availability. In the climate characteristics field, specify the climate type as available from atlases. Indicate the average temperature in January and in July, the average annual sunshine duration and the name and location of the closest weather station. For the map showing the immediate surroundings of the station, provide an original one. Indicate the scale and the North orientation. Use only a pencil to write something on the map. We will put the label «IDMP Station» ourselves. The Station section gives a list of the measuring and recording equipment, including manufacturers names and model numbers as well as details of calibration. The sampling interval indicates how often, you scan the output of the sensors. The recording interval indicates how often, you write the output on disk. If you are using shadow bands for diffuse measurements, provide their radius and width. For the detailed map of the station, all you need to do is a simple sketch of how your station has been setup and where the sensors are. Again indicate a scale on your sketch and the North orientation. In order to reduce the space taken by the obstruction information for all sensors, we have chosen to represent obstructions only for the global horizontal illuminance cell. This representation should be presented using azimuths on the x axis (going from 0/North to 90/East, 180/South...). On the y axis, the height of the obstructions above the horizon should be represented on a linear scale. Indicate whether these obstructions are trees or buildings, and whether the buildings are light or dark colored. In the comments field, you have additional space for the information which does not find a place anywhere else in this section. If you took part to the calibration plan started by TC 3.25, the Station section is also the place where you should show the results of the comparison between your sensors and CIE reference sensors. The output of a reference sensor should be shown on the x axis while the output of one of your sensor should be shown on the y axis. The scale should be linear. The Views section presents two pictures of the station. They should be chosen to show as much of the equipment as possible. They should also give an indication of what the obstructions look like. We recommend that you take these two pictures at the main sensor level using a wide angle lens. For each picture, provide comments indicating the direction towards which the picture is looking and what equipment can be seen on the picture. The Maintenance section indicates important modifications made to your IDMP station. These are informations that you feel other people should know. It could be: a change in the recording interval, or the periods of time when the station was not operating.
Data exchange format and policy Being a member of the IDMP network has its benefits but also its obligations: such as exchanging at least one year of data for free, with other members of the network. There is however, a one year delay period, to enable the collectors own research on his data and a two year delay period, during which data may only be disseminated to members of the network. Each member of the IDMP network is responsible for the dissemination of his data. Contact a member by e-mail or regular mail using the information provided in the Contacts section of the IDMP station description. A member receiving data from another member has to sign a «license agreement» which stipulates that he will not distribute the data to a third party. The IDMP guide gives a complete description of the format which is used to distribute the data. An important feature of this format is that the measured data is immediately followed by a flag which gives crucial information on the automatic and manual quality control diagnostics. Files are written with ASCII characters separated by white space separators. A file begins with comments preceded by a # sign and a header which describes what is in the file. The header uses measurement codes (3 digit numbers) which are given in the IDMP guide. Each record including the header, begins with the code of the station and ends with a «carriage return» character. Long records can be split by «carriage return» characters into groups of about 10 entry/diagnostic, although this is not recommended, if you want to use a spreadsheet program. The automatic quality control program called AQCCIE generates a file containing automatic quality control flags. The TIMElux program allows you to display the data and show its quality with different colors: red for questionable, orange for an inconclusive test, and green for validated. A fourth color is used when the data has not been tested. Frequency distributions, time series, fixed altitude time series can be generated on the basis of one minute, 5 minute averaged, or half hour averaged values. Further instructions are given in the IDMP guide. |